Question and answer effect evaluation method and system

Abstract

The present invention relates to a technical solution for a question-and-answer effect evaluation method and system, comprising: marking the quality of the answer based on the evaluation result of the answer, and generating a label for the question-and-answer data; inputting the question-and-answer data and the label of the question-and-answer data into an improved pre-trained language representation model for training; wherein, the first word vector corresponding to the question-and-answer data is obtained through the tag embedding layer, segmentation layer and position encoding layer of the improved pre-trained language representation model; and extracting the text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model, and inputting them into the classification layer of the improved pre-trained language representation model to obtain the answer quality evaluation result. The above solution improves the accuracy of the question-and-answer effect evaluation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a question-and-answer effect evaluation method and system. Background Art

[0002] In the question-and-answer effectiveness evaluation, the more closely the questions and answers match, the better the results of the question-and-answer effectiveness evaluation. The question-and-answer effectiveness evaluation has a wide range of applications. For example, government agencies have the service and responsibility to answer citizens' questions. Quantitatively evaluating the results of these services can not only determine whether these questions and answers have truly helped citizens, but also serve as an indicator for reviewing the agency's capabilities. The judgment of this system can help government agencies answer mission questions more standardizedly.

[0003] Existing technologies generally use manual methods to judge the degree of match between questions and answers. For example, technicians manually write a large number of matching rules and logic in advance, and use manual rules to judge the degree of match between questions and answers. In practice, it is found that with the increase in question and answer data, manual methods seem to be inadequate in today's era of big data. How to make accurate and fast judgments on large-scale question and answer data is a problem to be solved. Summary of the Invention

[0004] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art, and to provide a question-answering effect evaluation method and system that can accurately and efficiently evaluate the question-answering effect.

[0005] The technical solution of the present invention includes a question-and-answer effect evaluation method and system, and the question-and-answer effect evaluation method includes: marking the quality of the answer according to the evaluation result of the answer, and generating a label for the question-and-answer data; inputting the question-and-answer data and the label of the question-and-answer data into an improved pre-trained language representation model for training; wherein, the first word vector corresponding to the question-and-answer data is obtained through the tag embedding layer, segmentation layer and position encoding layer of the improved pre-trained language representation model; the text semantic features of the first word vector are extracted through the self-attention mechanism of the improved pre-trained language representation model, and input into the classification layer of the improved pre-trained language representation model to obtain the answer quality evaluation result.

[0006] According to the question-and-answer effect evaluation method, each question-and-answer data includes at least one answer, one of the answers is screened out as the first answer, and the quality of the answer is annotated based on the evaluation result of the first answer.

[0007] According to the question-answer effect evaluation method, the evaluation result of the answer is obtained, and the quality level of the answer is determined according to a preset threshold.

[0008] According to the question-answer effect evaluation method, the improved pre-trained language representation model is trained, including: the question-answer data is passed through the cascaded tag embedding layer and the segmentation layer to obtain a second word vector, after filling the second word vector, a third word vector representing the position information of the second word vector is obtained through a convolutional neural network.

[0009] According to the question-and-answer effect evaluation method, the training of the improved pre-trained language representation model includes: obtaining a fourth word vector from the question-and-answer data through the tag embedding layer, and obtaining the first word vector by calculating the sum of the second word vector, the third word vector, and the fourth word vector.

[0010] According to the question-answering effect evaluation method, the improved pre-trained language representation model is trained, including: selecting a Gaussian kernel as the kernel function in the self-attention mechanism, mapping the input first word vector to a Gaussian space, so as to learn the text semantic features of the isotropically distributed first word vector.

[0011] According to the question-and-answer effect evaluation method, before marking the quality of the answer based on the evaluation result of the answer, it also includes: screening out a preset number of the question-and-answer data based on the hotness of the topic corresponding to the question-and-answer data.

[0012] According to the question-answer effect evaluation method, the evaluation results of the answers include the number of likes, scores, and satisfaction or dissatisfaction patterns, numbers or text.

[0013] The technical solution of the present invention also includes a question-and-answer effect evaluation system for implementing any of the above methods, including a labeling module for labeling the quality of the answer based on the evaluation result of the answer and generating a label for the question-and-answer data; a training module for inputting the question-and-answer data and the label of the question-and-answer data into an improved pre-trained language representation model for training.

[0014] According to the question-answering effect evaluation system, the training module also includes: a word vector module, which is used to calculate the word vector through a tag embedding layer, a segmentation layer, or a position encoding layer; a text semantic feature module, which is used to select a Gaussian kernel as the kernel function in the self-attention mechanism, and map the input first word vector to a Gaussian space to learn the text semantic features of the isotropically distributed first word vector; an answer quality evaluation module, which is used to extract the text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model, and input them into the classification layer of the improved pre-trained language representation model to obtain the answer quality evaluation result.

[0015] The beneficial effects of the present invention are: evaluating question and answer data based on the improved pre-trained language representation model to improve the evaluation accuracy of the pre-trained language representation model; using CNN absolute position encoding to replace the traditional position encoding method, so that the pre-trained language representation model can learn the hidden position information of the text while having lower computational complexity; selecting a Gaussian kernel function to transform the word vector into an isotropic Gaussian space, renormalizing the input word vector distribution, and extracting semantic features with strong representation capabilities, thereby improving the evaluation accuracy of the pre-trained language representation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0017] Figure 1 Shown is a flowchart of a question-and-answer effect evaluation method according to an embodiment of the present invention.

[0018] Figure 2 Shown is a detailed flow chart of a question-and-answer effect evaluation method according to an embodiment of the present invention.

[0019] Figure 3 Shown is another flow chart of a question-and-answer effect evaluation method according to an embodiment of the present invention.

[0020] Figure 4 Shown is another flow chart of a question-and-answer effect evaluation method according to an embodiment of the present invention.

[0021] Figure 5 Shown is a structural diagram of a question-answering effect evaluation system according to an embodiment of the present invention.

[0022] Figure 6 Shown is a schematic diagram of a question-answering effect evaluation web service according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it should not be understood as a limitation on the scope of protection of the present invention.

[0024] In the description of the present invention, unless otherwise clearly defined, words such as “setting” should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0025] like Figure 1 As shown in , a question-answering effect evaluation method is Figure 1 The specific steps are as follows:

[0026] S100, based on the evaluation results of the answers, the quality of the answers is marked and labels for the question and answer data are generated;

[0027] S200, input the question and answer data and the labels of the question and answer data into the improved pre-trained language representation model for training; wherein, the first word vector corresponding to the question and answer data is obtained through the tag embedding layer, segmentation layer and position encoding layer of the improved pre-trained language representation model; the text semantic features of the first word vector are extracted through the self-attention mechanism of the improved pre-trained language representation model, and input into the classification layer of the improved pre-trained language representation model to obtain the answer quality evaluation result.

[0028] The following combination Figure 2 The flowchart shown is as follows: Figure 2 As shown, detailed implementation of the above steps is described in multiple embodiments.

[0029] For neural networks, conventional pre-trained models are trained using large datasets and are capable of extracting both shallow, basic features and deep, abstract features. However, these models may face risks such as non-convergence, suboptimal parameters, low accuracy, poor generalization, and overfitting. To avoid these drawbacks, conserve computing resources and time, and improve efficiency, the evaluation model training of the present invention is divided into two phases: pre-model training and fine-tuning model training.

[0030] The dataset of the present invention mainly includes a pre-training dataset and a fine-tuning dataset. Different datasets correspond to different training stages. The pre-training dataset corresponds to the training of the pre-model, and the fine-tuning dataset corresponds to the training of the fine-tuning model.

[0031] The pre-training dataset uses a third-party question-answering quality assessment dataset. For example, the present invention uses webtext2019zh as the pre-training dataset. Each question-answer data in the pre-training dataset contains the topic type, question description, and one or more replies. Each reply is labeled with the number of likes, reply ID, and respondent, as shown in Table 1.

[0032] In one embodiment, 160,000 government Q&A data collected in real scenarios are used as a fine-tuning dataset. Each Q&A data in the fine-tuning dataset contains information such as the type of appeal, the location of the incident, the title of the appeal, the question being consulted, the corresponding reply processing department, and evaluation, as shown in Table 2.

[0033] Table 1. Question and answer data

[0034]

[0035] S110, each question and answer data includes at least one answer, one answer is screened out as the first answer, and the quality of the answer is annotated according to the evaluation result of the first answer.

[0036] In one embodiment, a response can be randomly selected as the first response to the question. Since there may be multiple responses in each question and answer data in the pre-training data set, and different responses correspond to different quality levels, in order to make each question and answer data correspond to a quality level, a response can be randomly selected as the default response to the question, so that each question and answer data corresponds to a quality level.

[0037] S120, obtaining the evaluation result of the reply, and determining the quality level of the reply according to a preset threshold, specifically including:

[0038] S121, uniformly convert the evaluation results into digital representation, set a threshold, and determine the quality level of the response based on the set threshold. The evaluation results in the question and answer data include the number of likes, ratings, and satisfaction patterns, numbers, or text.

[0039] The evaluation results are generally the feedback after answering the questions. The evaluation results can be in the form of numbers, for example, selecting a number from 1-100, or 1-10, or 1-5 to score the answer. The evaluation can be in the form of pictures, such as likes, stars, etc.; the evaluation can be in the form of text, such as yes / no, or satisfied / unsatisfied, etc.

[0040] Table 2. Question and answer data

[0041]

[0042] In one embodiment, when the evaluation is in text form, for example, yes / no, or satisfied / unsatisfied, the evaluation results are uniformly converted into digital representations. Discrete numbers can be used to represent yes / no, satisfied / unsatisfied, for example, 1 represents no or unsatisfied; 2 represents yes or satisfied.

[0043] In one embodiment, the evaluation is in the form of a pattern, such as a like, a star, etc., and the evaluation results are uniformly converted into digital representations and can be counted based on the number of likes or stars.

[0044] The classification types can be adjusted according to actual conditions. The present invention does not enumerate all classification situations one by one. For example, the present invention divides the reply quality into two categories: high quality and poor quality. Among them, if the number of likes exceeds a threshold value h, the reply quality is considered to be high, and if it is lower than the threshold value h, the reply quality is considered to be poor. The threshold value h can be selected within a certain range. For example, the threshold value h can be selected between 1-50. When the threshold value h is 3, the number of likes exceeds 3, the reply quality is considered to be high, and if it is lower than 3, the reply quality is considered to be poor.

[0045] S130: Filter out a preset number of the question and answer data according to the digital representations of the evaluation results in the question and answer data.

[0046] The larger the numerical representation of the evaluation result, the more recognized the answer is by netizens, and the more likely it is to be a high-quality answer. Depending on the popularity of the question, it is difficult to uniformly classify the quality of the answer. For example, when browsing hot questions, you will often see some answers with more than 100 likes, while for less popular questions, the number of likes is often very small, almost zero. In order to obtain a more accurate evaluation model, a preset amount of the question and answer data is filtered out based on the numerical representation of the evaluation results in the question and answer data.

[0047] In one embodiment, the first threshold value of the digital representation of the evaluation result is H, and the second threshold value of the digital representation of the evaluation result is L. The digital representation of the evaluation result exceeding the first threshold value H or the digital representation of the evaluation result less than the second threshold value L is screened out. For example, the first threshold value of the digital representation of the evaluation result is H, which is 50, and the second threshold value of the digital representation of the evaluation result is L, which is 1. The digital representation of the evaluation result exceeding 50 or the digital representation of the evaluation result less than 1 is screened out.

[0048] S140: Desensitize sensitive information in the question-and-answer data. For example, the conversation content or other features in the fine-tuning dataset may involve privacy, information security, etc., and the fine-tuning dataset needs to be desensitized to remove information related to privacy, information security, etc.

[0049] S150, dividing the data set into a training set, a test set, and a validation set according to a preset ratio. The present invention divides the data set into a training set, a test set, and a validation set according to a preset ratio to train, test, and validate the evaluation model.

[0050] The pre-training data set is divided into a pre-training set, a pre-test set, and a pre-verification set according to a ratio. The ratio can be set reasonably. The present invention sets the ratio to 8:1:1, where the pre-training set, pre-test set, and pre-verification set are 4.12 million, 515,000, and 515,000 respectively.

[0051] The fine-tuning dataset is divided into a fine-tuning training set, a fine-tuning test set, and a fine-tuning validation set according to a ratio. For example, the present invention sets the ratio to 8:1:1, where the fine-tuning training set, the fine-tuning test set, and the fine-tuning validation set are 128,000, 16,000, and 16,000 respectively.

[0052] Since the Transformer network uses a self-attention mechanism and cannot capture position information in the sequence, the present invention adopts an improved pre-trained language representation model to extract features and evaluate the response effect. Unlike the traditional unidirectional language model or the method of shallowly splicing two unidirectional language models for pre-training, the language representation model of the present invention can fuse the deep bidirectional language representation of the left and right context information and extract more comprehensive features of the relationship between words. Among them, the improved pre-trained language representation model includes input, tag embedding layer, segmentation layer, position encoding layer and classifier.

[0053] S200 inputs the question-and-answer data and the labels of the question-and-answer data into an improved pre-trained language representation model for training; wherein, a first word vector corresponding to the question-and-answer data is obtained through the tag embedding layer, segmentation layer, and position encoding layer of the improved pre-trained language representation model; the text semantic features of the first word vector are extracted through the self-attention mechanism of the improved pre-trained language representation model, and input into the classification layer of the improved pre-trained language representation model to obtain the answer quality assessment result. Specifically including:

[0054] S210, the question and answer data is embedded in the label layer to obtain the fourth word vector, and the first word vector is obtained by calculating the sum of the second word vector, the third word vector, and the fourth word vector, such as Figure 3 As stated,

[0055] Specifically include:

[0056] S211, tokenization of question and answer data. Before outputting multiple pairs of question and answer data x[n] to the tag embedding layer, they must be tokenized first. Text tokenization is mainly performed through word segmentation through WordPiece. For example, "AlphaGo" is divided into "AlphaGo", "Fargo", and "Dog". In addition, two special tags "[CLS]" and "[SEP]" will be inserted at the beginning and end of the tokenization result, where "[CLS]" is also called the "classification head". It is a symbol without obvious semantic information. The corresponding vector of "[CLS]" can be used as the semantic representation of the entire sentence, so as to be used for downstream classification tasks. "[SEP]" is both a symbol that separates sentences and a symbol that indicates the end of input. For example, the result of tokenizing a pair of question and answer texts x[1] is as follows:

[0057] [CLS] “A”, “law”, “dog”, “only”, “can”, “play”, “chess”, “does”, “?”, “can”, “write”, “small”, “say”, “do”, “?”[SEP] “I”, “believe”, “A”, “law”, “dog”, “can”, “superior”, “surpass”, “human”, “kind”[SEP].

[0058] S212: The question-and-answer data is passed through the token embedding layer to obtain a fourth word vector. The token embedding layer is used to convert each word into a vector of fixed dimension. For example, in the present invention, each character is converted into a 768-dimensional vector representation. The 30 tokens in the above example are converted into a (30, 768) matrix or a (1, 30, 768) tensor, where one token represents one character.

[0059] S213: The question-answer data is passed through the cascaded tag embedding layer and the segmentation layer to obtain a second word vector. The segmentation layer is used to assist the language representation model in distinguishing the vector representations of the two sentences in a sentence pair. For example, in the above example, the segmentation layer can be represented by a (n, 768) matrix or a (1, n, 768) tensor, where n is 30.

[0060] S214: After filling the second word vector, a third word vector representing the position information of the second word vector is obtained through a convolutional neural network.

[0061] The position encoding layer is used to learn the sequential properties of the input. Unlike the traditional pre-trained language representation model that uses a single-layer fully connected network for mapping position encoding, the present invention has made improvements in the position encoding layer. By taking advantage of the CNN convolutional neural network's strong representation ability in position information, CNN is used as the encoder of position information, further reducing the computational workload and being able to comprehensively and accurately learn the position information of the input text. In the above example, the position encoding layer can be represented by a (n, 768) matrix or a (1, n, 768) tensor, where n is 30.

[0062] The position encoder includes an input layer, a convolution layer, a pooling layer, and a fully connected layer, wherein the input layer of the position encoding layer is a matrix of size (n, d). Since the present invention is mainly based on text data for feature extraction, only the width is convolved and the height is not convolved, so a one-dimensional convolution kernel is used for convolution. For example, the convolution layer of the present invention uses a one-dimensional convolution kernel of sizes (1, n / 3), (1, n / 4), and (1, n / 5) for convolution to obtain j feature maps, wherein the number of each convolution is k, j = 3*k(nn / 3+1)+3*k(nn / 4+1)+3*k(nn / 5+1) , perform zero-padding operation on the above feature map, that is, add 0 at the beginning and end, and then use the same set of convolution kernels for convolution to obtain a new feature map; the pooling layer uses maxpooling to perform pooling operation on the new feature map to obtain a feature vector of (1, 3k), and the fully connected layer uses a fully connected layer of size (3k, d) to map to obtain a position encoding vector of (1, d), where the activation function uses sigmoid. Finally, the position encoding of (1, d) is expanded to an embedding matrix of size (n, d) and added to the input layer matrix (n, d) to embed the position information.

[0063] In the above example, the position encoding layer can be represented by a (30, 768) matrix. The input of the position encoding layer is a (30, 768) matrix. The convolution layer sets the one-dimensional convolution kernels of size (1, 10), (1, 7), and (1, 6) for convolution to obtain 360 feature maps. The number of convolutions is k = 2, j = 360, and the zero-padding operation is performed on the feature maps, that is, adding 0 at the beginning and end, and convolution is performed again. The sizes of the convolution kernels are also (1, 10), (1, 7) and (1, 6). ), (1, 6), and obtain a new feature map; the pooling layer uses maxpooling to perform pooling operation on the new feature map, and splices it to obtain a feature vector of (1, 6); the fully connected layer uses a matrix of size (6, 768) to map the feature vector to obtain a position encoding vector of (1, 768), where the activation function uses sigmoid; finally, the position encoding vector of (1, 768) is expanded to an embedding matrix of size (30, 768) and added to the input matrix (30, 768) to realize the embedding of position information.

[0064] S215: Calculate the sum of the second word vector, the third word vector, and the fourth word vector to obtain a first word vector, where the first word vector is denoted as x′[n].

[0065] The use of CNN as the encoder of position information mainly utilizes the implicit position encoding mechanism in CNN. It learns that implicit position information plays an important role in text processing and can accurately classify the quality of responses. In addition, it has lower computational complexity than the relative position information encoding of traditional pre-trained language representation models.

[0066] S220: Extract text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model.

[0067] The text semantic features of the first word vector are extracted by the self-attention mechanism of the improved pre-trained language representation model, which can integrate the deep bidirectional language representation of the left and right context information and extract more comprehensive features of the relationship between words. The self-attention mechanism network includes a multi-head self-attention sublayer, a feedforward neural network sublayer and a residual connection layer. For example, the present invention uses the self-attention mechanism to extract the text semantic features x of the first word vector x′[n] by processing it with a Transformer-based feature extractor. * [n], such as Figure 4 As shown, specifically including:

[0068] S221, select a Gaussian kernel as the kernel function in the self-attention mechanism, map the input first word vector to a Gaussian space, and learn the text semantic features of the isotropically distributed first word vector.

[0069] Because the word vectors input to the feature extractor have anisotropy in their space, the word vectors learned by the Transformer are semantically non-smooth and cannot represent semantics in certain places in the space, resulting in low accuracy in response quality classification. To eliminate the anisotropy of word vectors, the present invention uses a Gaussian kernel instead of the linear kernel in the self-attention mechanism, mapping the input word vectors to a Gaussian space. This results in a smooth, isotropic Gaussian distribution of the mapped word vectors. The extracted features can better represent semantics, thereby improving the accuracy of response quality classification.

[0070] In step S222, the multi-head self-attention sublayer converts the input x′[n] into a matrix attention feature vector A[n] through the self-attention mechanism. The input of the multi-head self-attention is first transformed with a Gaussian kernel and then fed into a scaled dot product. This is repeated multiple times. The results of these scaled dot products are then concatenated and linearly transformed to obtain the value used as the multi-head output. This allows the feature extractor to learn more about word-to-word information in different subspaces.

[0071] S223, add the first word vector to the matrix attention feature vector and perform layer standardization to obtain the output vector of the attention sub-layer after standardization. For example, add the input x′[n] to the attention feature vector A[n] and perform layer standardization to obtain the vector B[n], wherein, in a network block of the encoder, it is composed of a multi-head self-attention sub-layer and a feedforward neural network sub-layer, and the sub-layer output is added to the sub-layer input and then normalized as the sub-layer normalized output, as shown in formula (1);

[0072] Output=input+sublayer(input) (1)

[0073] Among them, Output represents the standardized output of the sublayer, input represents the input of the sublayer, and sublayer(input) is the output of the sublayer.

[0074] S224, input the normalized output vector of the attention sublayer into the feedforward neural network to obtain a feedforward vector, add the normalized output vector of the attention sublayer and the feedforward vector, and then perform layer normalization to obtain the text semantic feature. For example, the normalized output vector B[n] of the attention sublayer is input into the feedforward neural network to obtain the feedforward vector C[n], add the normalized output vector B[n] of the attention sublayer and the feedforward vector C[n], and then perform layer normalization to extract the text semantic feature x * [n].

[0075] In one embodiment, N=1, a 3-layer Transformer network is used to construct a text semantic feature extraction model. Therefore, there are 3 attention sub-layers plus 3 feed-forward neural network sub-layers, a total of 6 sub-layers. The input is a (30,768) matrix, and the output of the extractor is the text semantic feature x * The dimension of [n] is 30*768, where the first feature dimension of the semantic feature is CLS and the last feature dimension is SEP.

[0076] The deep residual network consists of n densely connected layers and concatenation layers, connected via residual connections. The densely connected layers are used to map the word vectors of (n-2, d) to one-dimensional feature vectors of (n-2, 1). The concatenation layer concatenates the cls vectors to the one-dimensional feature vectors of (n-2, 1) to form a new feature vector (n-1, 1). The first and last feature dimensions of the semantic features are removed to form a new word vector (n-2, d), which is used as the input to the deep residual network. The input word vectors are regularized using dropout, and RELU is used as the activation function. The final output is a one-dimensional feature vector (n-2, 1). The concatenation layer concatenates the cls vectors along the axis to the one-dimensional feature vectors of (n-2, 1) to form a new feature vector (n-1, 1).

[0077] S230, the text semantic features of the first word vector are input into the classification layer of the improved pre-trained language representation model to obtain a response quality assessment result.

[0078] The text semantic features are input into the fully connected layer to classify the response quality. The classification layer includes a fully connected layer. The new feature vector (n-1,1) is used as the input of the classification layer. It is classified through a softmax fully connected layer. The response quality classification result is the result of the question and answer response performance evaluation.

[0079] The present invention adopts an improved pre-trained language representation model to evaluate the question-answering effect, and adopts CNN absolute position encoding to replace the traditional position encoding method, which can learn the hidden position information of the text while having low computational complexity. It selects a Gaussian kernel function to transform the word vector into an isotropic Gaussian space, renormalizes the input word vector distribution, and extracts semantic features with strong representation ability, thereby improving the evaluation accuracy of the pre-trained language representation model.

[0080] In a second aspect, an embodiment of the present invention further provides a question-answering effect evaluation system, such as Figure 5 Shown, including:

[0081] The annotation module is used to annotate the quality of the answers based on the evaluation results and generate labels for the question and answer data;

[0082] The training module is used to input question-answering data and labels of the question-answering data into the improved pre-trained language representation model for training.

[0083] The training module also includes:

[0084] The word vector module is used to calculate the word vector through the token embedding layer, segmentation layer, or position encoding layer;

[0085] A text semantic feature module, configured to select a Gaussian kernel as a kernel function in a self-attention mechanism, and map the input first word vector to a Gaussian space to learn text semantic features of the first word vector with an isotropic distribution;

[0086] The answer quality assessment module is used to extract the text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model, and input them into the classification layer of the improved pre-trained language representation model to obtain the answer quality assessment result.

[0087] In one embodiment, if Figure 6As shown, the Q&A effectiveness evaluation system provides an inference service, which consists of a backend application and a web page display component. Users enter two sentences for comparison on the web page to the backend, which evaluates the Q&A effectiveness based on the system and returns the model results to the web frontend for display.

[0088] For example:

[0089] Text 1: "What's the best way for lazy people to lose weight?"

[0090] Text 2: "I don't know what level of laziness this is. If it's just laziness that means not wanting to exercise, then you can achieve weight loss by changing your eating habits."

[0091] Evaluation result: low quality

[0092] The question-answering effect evaluation system of the present invention can enable a computer to automatically identify high-quality questions and answers, and automatically and on a large scale perform efficient and accurate quality evaluation of the information in the question-answering system.

[0093] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or executed by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can be run on a programmed application-specific integrated circuit.

[0094] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that can be executed by one or more processors.

[0095] Further, the methods can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. In addition, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0096] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data that is stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0097] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the technical field without departing from the spirit of the present invention.

Claims

1. A question-answering effect evaluation method, characterized in that: The following steps are involved: Based on the evaluation results of the answers, the quality of the answers is marked to generate labels for the question and answer data; Inputting the question-answer data and the labels of the question-answer data into an improved pre-trained language representation model for training; wherein the first word vector corresponding to the question-answer data is obtained through the tag embedding layer, the segmentation layer, and the position encoding layer of the improved pre-trained language representation model, wherein the improved pre-trained language representation model includes a feature extractor of a Transformer; Extracting text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model, and inputting the features into the classification layer of the improved pre-trained language representation model to obtain a response quality assessment result; Training the improved pre-trained language representation model includes: selecting a Gaussian kernel as a kernel function in the self-attention mechanism, mapping the input first word vector to a Gaussian space, and learning text semantic features of the isotropically distributed first word vector; The training of the improved pre-trained language representation model includes: obtaining a second word vector from the question and answer data through the cascaded tag embedding layer and the segmentation layer, filling the second word vector, and then obtaining a third word vector representing position information of the second word vector through a convolutional neural network; The training of the improved pre-trained language representation model includes: obtaining a fourth word vector from the question and answer data through the tag embedding layer, and obtaining the first word vector by calculating the sum of the second word vector, the third word vector, and the fourth word vector.

2. The question-answering effect evaluation method according to claim 1, characterized in that: Each piece of the question-and-answer data includes at least one answer, one of the answers is screened out as a first answer, and the quality of the answer is marked based on the evaluation result of the first answer.

3. The question-answering effect evaluation method according to claim 1, characterized in that: The evaluation results of the responses are obtained, the evaluation results are uniformly converted into digital representations, and the quality level of the responses is determined according to a preset threshold.

4. The question-answering effect evaluation method according to claim 3, characterized in that: Before marking the quality of the answer based on the evaluation result of the answer, the method further includes: screening out a preset number of the question and answer data based on the digital representation of the evaluation result in the question and answer data.

5. The question-answering effect evaluation method according to claim 1, characterized in that: The evaluation results of the reply include the number of likes, ratings, and satisfaction or dissatisfaction patterns, numbers or text.

6. A question-answering effect evaluation system, characterized in that: The method for evaluating the question-answering effect according to claim 1 includes: A labeling module is used to label the quality of the answer based on the evaluation results of the answer and generate labels for the question and answer data; A training module, configured to input the question-and-answer data and the labels of the question-and-answer data into an improved pre-trained language representation model for training; The training module also includes a text semantic feature module, which is used to select a Gaussian kernel as the kernel function in the self-attention mechanism, and map the input first word vector to a Gaussian space to learn the text semantic features of the isotropically distributed first word vector.

7. The question-answering effect evaluation system according to claim 6, characterized in that: The training module also includes: The word vector module is used to calculate the word vector through the token embedding layer, segmentation layer, or position encoding layer; The answer quality assessment module is used to extract the text semantic features of the first word vector through the self-attention mechanism of the improved pre-trained language representation model, and input them into the classification layer of the improved pre-trained language representation model to obtain the answer quality assessment result.

Images