Model generation method, question and answer quality judgment method, device, equipment and medium

Abstract

The disclosed embodiments disclose a model generation method, an answer quality judgment method, an apparatus, a device and a medium, wherein the model generation method comprises: obtaining a preset number of question-answer pair sample sets, and analyzing the sample data in each question-answer pair sample set to obtain a word vector; inputting the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample; determining the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculating the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability; generating a question-answer pair quality judgment model when the loss function result meets the preset conditions. The disclosed embodiments solve the problem of low accuracy in question-answer quality judgment when the question-answer pair content is large or complex, and realize the judgment of question-answer quality when the question-answer content is complex, thereby improving the accuracy of question-answer quality judgment.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer applications, and in particular to a model generation method, an answer quality judgment method, an apparatus, a device, and a medium. Background Art

[0002] With the development of the Internet, many interactive question-and-answer platforms have emerged, from which users can acquire or share knowledge. The platform reviews the answers to the questions and displays the approved answers on the question-and-answer platform.

[0003] In the existing technology, the quality of question-answer pairs is usually judged from the perspectives of artificial rules, feature engineering, similarity matching, semantic coherence, etc. However, among the reviewed question-answer pairs with more content and more complex question-answer text pairs, there will still be a large number of low-quality answers, and there will be cases where the answers are irrelevant to the questions. The accuracy of question-answer quality judgment needs to be further improved.

[0004] Public Content

[0005] The embodiments of the present disclosure provide a model generation method, a question and answer quality judgment method, an apparatus, a device and a medium to enable a judgment on the question and answer quality when the question and answer content is complex, thereby improving the accuracy of the question and answer quality judgment.

[0006] In a first aspect, an embodiment of the present disclosure provides a model generation method, the method comprising:

[0007] Obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector;

[0008] Input the word vector into the pre-trained language model to obtain a representation vector for each question-answer pair sample;

[0009] Determine the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculate the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability;

[0010] When the loss function result meets the preset conditions, a question-answer pair quality judgment model is generated.

[0011] In a second aspect, the embodiments of the present disclosure further provide a method for judging the quality of a question and answer, the method comprising:

[0012] Obtaining a question-answer pair text to be judged, and generating a question-answer pair quality judgment model based on the model generation method described in any embodiment of the present disclosure;

[0013] The question-answer pair text to be judged is input into the question-answer pair quality judgment model to obtain the quality judgment result of the question-answer pair to be judged.

[0014] In a third aspect, the present disclosure also provides a model generation device, the device comprising:

[0015] A first sample analysis module is used to obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector;

[0016] A second sample analysis module is used to input the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample;

[0017] A third sample analysis module determines the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculates the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability;

[0018] The model generation module is used to generate a question-answer pair quality judgment model when the loss function result meets the preset conditions.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a question and answer quality judgment device, the device comprising:

[0020] A text and model acquisition module, used to acquire the question-answer pair text to be judged, and generate a question-answer pair quality judgment model based on the model generation method described in any embodiment of the present disclosure;

[0021] The question and answer quality judgment module is used to input the question and answer pair text to be judged into the question and answer pair quality judgment model to obtain the quality judgment result of the question and answer pair to be judged.

[0022] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0023] one or more processors;

[0024] A memory for storing one or more programs;

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the model generation method or question and answer quality judgment method as described in any embodiment of the present disclosure.

[0026] In a sixth aspect, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the model generation method or the question and answer quality judgment method as described in any embodiment of the present disclosure is implemented.

[0027] The disclosed embodiment uses a set containing multiple question-answer pair samples as a sample input unit, performs semantic analysis on the question-answer pair samples in the set, and then calculates the probability value of the predicted label of each question-answer pair in a set being the true label, and selects the question-answer pair with the highest probability value as the preferred model training data, adjusts the parameters of the model, and generates a question-answer pair quality judgment model when the loss function result meets the preset conditions. The problem of low accuracy in judging the question-answer quality of complex question-answer pairs in the prior art is solved, and the stability of the generated question-answer pair quality judgment model is enhanced, so that the question-answer pair quality judgment model makes higher accuracy judgments when the question-answer content is complex. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0029] Figure 1 is a flow chart of the model generation method in the first embodiment of the present disclosure;

[0030] Figure 2 is a flow chart of the model generation method in the second embodiment of the present disclosure;

[0031] Figure 3 is a structural diagram of the question-answer quality judgment model in the second embodiment of the present disclosure;

[0032] Figure 4 is a flow chart of the question and answer quality judgment method in Embodiment 3 of the present disclosure;

[0033] Figure 5 is a schematic diagram of the structure of the model generating device in the fourth embodiment of the present disclosure;

[0034] Figure 6 This is a schematic diagram of the structure of the question and answer quality judgment device in the fifth embodiment of the present disclosure.

[0035] Figure 7 It is a structural diagram of an electronic device in Embodiment 6 of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0038] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0039] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0042] Embodiment 1

[0043] Figure 1 A flowchart of a model generation method provided in the first embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to the case where a question and answer quality judgment model is trained based on a large amount of question and answer pair data. The method can be implemented by a model generation device, and specifically can be implemented by software and / or hardware in an electronic device.

[0044] like Figure 1 As shown, the model generation method provided in the embodiment of the present disclosure includes the following steps:

[0045] S110, obtaining a preset number of question-answer pair sample sets, and analyzing the sample data in each question-answer pair sample set to obtain a word vector.

[0046] Specifically, the question-answer pair sample data refers to the question data and answer data in question-answering, knowledge sharing and other platforms. There may be multiple answers to a question, and a question will have multiple answers. Then, the question and each answer constitute multiple question-answer pair data. The question-answer pair sample data can also be questions in books and corresponding answers, for example, after-class questions in Chinese textbooks. The answers to after-class questions include the standard answers in the lesson plan and the corresponding answers given by each student. A question and a corresponding answer constitute a question-answer pair sample.

[0047] The question-answer pair samples in this embodiment are samples that have been pre-labeled, and each question-answer pair sample has a label that the answer is not the question or a label that the answer is the question. The data labeling method includes manual labeling, that is, manually identifying and determining the label corresponding to a question-answer pair sample, or obtaining the label corresponding to the question-answer pair from a standard and reliable platform.

[0048] Furthermore, a sample set contains question-answer pair samples with the same label. The number of question-answer pair samples in the set can be determined based on the amount of computation in the model generation process and the computing power of the hardware, or a comprehensive value based on indicators such as model training efficiency. When analyzing the question-answer pair sample data, multiple question-answer pair samples in a sample set are analyzed at the same time, similar to batch synchronization processing. The specific data analysis process includes: segmenting each question-answer pair sample, for example, "Is today Arbor Day? No, it will be tomorrow", after semantic segmentation, we can get "Today / Is / Arbor Day / Is / Not / Tomorrow / It will be". Then, based on the segmentation results, the vocabulary of the pre-trained language model is queried to initialize the representation vector of each word, for example, the representation vector corresponding to "today" is "XXXXX".

[0049] Among them, the pre-trained language model adopts the method of self-supervised learning. By sending a large amount of unsupervised text into the model for learning, a general pre-trained model can be obtained, which is beneficial to the subsequent text language processing process. At present, the vocabulary corresponding to the pre-trained language model is a relatively mature word vector database with rich content. Mapping the word segmentation results to the vocabulary of the pre-trained language model can obtain the help of high-frequency co-occurrence world knowledge, which is beneficial to the understanding of the semantics of question and answer samples and the subsequent question and answer quality judgment.

[0050] Pre-trained language models include autoregressive language models and autoencoding language models. A suitable pre-trained language model can be selected according to the characteristics of the specific application scenario. In this embodiment, the pre-trained language model can be applied to the quality judgment of question and answer. The goal of the pre-trained language model is to use large-scale unlabeled corpus training to obtain a Representation of the text containing rich semantic information, that is, the semantic representation of the text, and then fine-tune the semantic representation of the text in a specific natural language processing task, and finally apply it to the natural language processing task. In this embodiment, the natural language processing task is to determine whether the question and answer pair is an answer rather than a question.

[0051] S120: Input the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample.

[0052] After mapping the molecular results of the question-answer pair samples to the pre-trained language model vocabulary in the above steps, the word vectors of each Chinese word in the word segmentation results can be obtained. Then, the word vector of each question-answer pair sample is input into the pre-trained language model to obtain the semantic representation vector of the question-answer pair sample.

[0053] In this embodiment, the question and answer in the question-answer pair are spliced ​​together and processed as a whole. Compared with modeling the question and answer separately, the effective vocabulary dependency between the question and the answer can be better captured, and the judgment of the more complex situation of answering the wrong question can be more accurate. Specifically, "complex situation of answering the wrong question" refers to the situation when the question text and the answer text have a strong correlation. In this case, it is difficult for the prior art to determine whether there is a phenomenon of answering the wrong question. For example: What is the effect of black rice? Purple rice is a variety of rice, belonging to the glutinous rice category, divided into purple japonica and purple glutinous rice. Generally speaking, purple glutinous rice is black glutinous rice; and black rice is a processed product of black rice, belonging to the japonica rice category, with two types of grain types, indica and japonica, and black rice is non-glutinous rice.

[0054] S130. Determine the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculate the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability.

[0055] Specifically, the representation vector of each question-answer pair sample in the set can be input into a fully connected neural network whose activation function is the Softmax function. The probability that the predicted label of each question-answer pair sample is the same as the true label is calculated through the fully connected neural network.

[0056] Furthermore, the probability values ​​corresponding to each question and answer pair sample in a question and answer pair sample set can be sorted, and the question and answer pair corresponding to the maximum probability value can be used as a preferred model training sample to adjust the parameters of the neural network. This is because when labeling the question and answer pair sample data, the labeling cost is high and the quality is difficult to control. Whether many questions are answered irrelevantly requires subjective judgment. Therefore, in the training data that answers irrelevant questions, it cannot be guaranteed that the labeled labels are completely correct, and there will be a large amount of noise data, that is, incorrectly labeled labels. In this embodiment, by selecting the question and answer pair sample corresponding to the maximum probability value of the predicted label as the labeled label from multiple question and answer pair samples to calculate the loss function, and then adjusting the model parameters, the model effect can be improved, and effective noise reduction can be performed in the model training stage (i.e., the model generation process).

[0057] S140. When the loss function result meets the preset conditions, a question-answer pair quality judgment model is generated.

[0058] The process of model generation is to continuously adjust the model parameters through sample learning to optimize the model function. Specifically, in this embodiment, the loss function is calculated by predicting the question-answer pair sample with the highest confidence level until the loss function converges to the pre-set numerical conditions, and the model parameters are determined, that is, the question-answer pair quality judgment model is generated.

[0059] The technical solution of the disclosed embodiment uses a set containing multiple question-answer pair samples as a sample input unit, performs semantic analysis on the question-answer pair samples in the set, and then calculates the probability value of the predicted label of each question-answer pair in a set being the true label, and selects the question-answer pair with the highest probability value as the preferred model training data, adjusts the parameters of the model, and generates a question-answer pair quality judgment model when the loss function result meets the preset conditions. The problem of low accuracy in judging the question-answer quality of complex question-answer pairs in the prior art is solved, and the stability of the generated question-answer pair quality judgment model is enhanced, so that the question-answer pair quality judgment model makes higher accuracy judgments when the question-answer content is complex.

[0060] Embodiment 2

[0061] Based on the above embodiments, this embodiment further describes the model generation process from the perspective of the structure of the model, which belongs to the same inventive concept as the model training method proposed in the above embodiments. For technical details not fully described in this embodiment, please refer to the above embodiments.

[0062] Figure 2 A flow chart of a model generation method provided in the second embodiment of the present disclosure is shown. The model generation method provided in the embodiment of the present disclosure includes the following steps:

[0063] S210. Input a set of question-answer pair samples containing a preset number of questions in units of groups through the input layer of the model to be trained.

[0064] The input layer of the model to be trained is used to input the processing object of the model to be trained. In this embodiment, the model to be trained is a question and answer quality judgment model, and the object processed by the model is the question and answer text data. In the training process of the model, a group of question and answer pair samples is taken as a unit of training samples, and the question and answer pairs in each group are question and answer pairs with consistent labels. The number of question and answer pairs in each group of samples can be the same or different, and the specific number of question and answer pairs can be determined according to hyperparameters. Among them, hyperparameters refer to parameters whose values ​​are set before starting the learning process, not parameter data obtained through model training. Generally, it is necessary to optimize the hyperparameters to improve the performance and effect of model training.

[0065] For example, in Figure 3 In the model structure diagram shown, the input layer inputs a set of question-answer pairs labeled "irrelevant answers", such as "What are the effects of black rice? Purple rice is... rice" and "What are the effects of black rice? Black rice nourishes yin and strengthens the kidney..." and so on.

[0066] S220. Perform semantic word segmentation on each of the question-answer pairs through the representation layer of the model to be trained, and determine the representation vector of each word in the word segmentation result.

[0067] The representation layer of the model to be trained is used to initialize the input question-answer pair text into the representation vector of each word in the question-answer pair text, in order to input the question-answer pair text into the pre-trained language model in a format that matches the pre-trained language model.

[0068] Specifically, first, we segment the samples for each question and answer; then, we query the vocabulary of the pre-trained language model based on the segmentation results to initialize the representation vector of each word. Through the mapping between words and vocabulary, we get Figure 3 In the figure, the word vector representation of the question-answer pair text shown in the representation layer. The first token of each sequence is always a special classification embedding [CLS], where one token represents one word vector; when sentence pairs are packaged into a sequence, they are often separated by a special token [SEP].

[0069] S230. Calculate the representation vector of each question-answer pair sample through the pre-trained language model encoding layer of the model to be trained.

[0070] The encoding layer of the pre-trained language model is the Encoder in the Transformer layer. The Transformer layer is formed by stacking several encoders and decoders, which are used to convert the input corpus into feature vectors. The input corpus is the word vector representation of each question-answer pair sample, and the feature vector is the feature vector that can represent the text semantics of the corresponding question-answer pair.

[0071] S240. In the prediction layer of the model to be trained, the likelihood probability that the predicted label of each question-answer pair sample is the correct label is determined based on the representation vector, and the target question-answer pair sample is selected through a multi-instance learning mechanism to calculate the loss function.

[0072] The prediction layer is used to predict the sample labels of each question and answer pair and obtain the predicted labels. Specifically, a fully connected layer is superimposed on the pre-trained language model to calculate the probability of the predicted label being a certain type of label, and the calculation result meets the requirements of the loss function.

[0073] In the process of model training, that is, the process of adjusting the parameters of the model to be trained. In this embodiment, a multi-instance learning mechanism is adopted to sort the probability values ​​corresponding to each question and answer pair sample in a question and answer pair sample set, and the question and answer pair corresponding to the maximum probability value is used as a preferred model training sample to adjust the parameters of the model to be trained. This is because when labeling the question and answer sample data, the labeling cost is high, and the quality is difficult to control. Whether many questions are answered irrelevantly requires subjective judgment. Therefore, it cannot be guaranteed that the labeled labels in the training data that are not answered are completely correct, and there will be a large amount of noise data, that is, incorrectly labeled labels. In this embodiment, by selecting the question and answer pair sample corresponding to the maximum probability value of the predicted label as the labeled label in multiple question and answer pair samples to calculate the loss function, and then adjusting the model parameters, the model effect can be improved, and effective noise reduction can be performed in the model training stage (that is, the model generation process).

[0074] like Figure 3 As shown, in the prediction layer, the input is the vector representation of the question-answer pair output by the encoding layer of the pre-trained language model (such as r t , r m ), the vector representation of the question-answer pair is input into the fully connected network to obtain the probability value of the predicted label of the question-answer pair being the true label (such as p t , p m). It should be noted here that in the model training stage, the number of question-answer pair representation vectors input into the fully connected network is multiple, and the specific number is consistent with the number of question-answer pairs in a set of sample sets input into the input layer, and the corresponding probability values ​​are also multiple. Then, the representation vector of the sample pair corresponding to the maximum value in the probability value is selected to calculate the loss function, adjust the model parameters, and finally obtain a trained question-answer quality judgment model. In the process of model application, in most cases, only one question-answer pair is judged at a time, that is, only one question-answer pair’s final representation vector is input into the fully connected network to obtain the model output result.

[0075] S250. When the loss function result meets the preset conditions, the training of the model to be trained is completed, and a question-answer pair quality judgment model is obtained.

[0076] When the parameters of the model to be trained can be the loss function convergence, and meet the preset conditions, the training process of the model to be trained is completed, and the question-answering quality judgment model is obtained. When there is a question-answer pair that needs to be judged whether it is an irrelevant answer, the question-answer pair can be input into the question-answering quality judgment model to obtain the label of the question-answer pair, that is, "irrelevant answer" or "exact answer".

[0077] The technical solution of the disclosed embodiment sets an input layer, a representation layer, a pre-trained language model conversion layer, and a prediction layer for the question-answer quality judgment model to be trained, gradually processes the input question-answer pair text, obtains the prediction label of the question-answer pair, and ensures the stability of the question-answer quality judgment model through a multi-instance learning mechanism in the prediction layer, so that the accuracy of the model output result is improved. The problem of low accuracy of question-answer quality in judging complex question-answer pairs in the prior art is solved, and the judgment of question-answer quality is made when the question-answer content is complex, thereby improving the accuracy of question-answer quality judgment.

[0078] Embodiment 3

[0079] Figure 4 A flowchart of a question and answer quality judgment method provided in the third embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to judging whether the answer to a question is irrelevant to the question. The method can be implemented by a question and answer quality judgment device, and can be specifically implemented by software and / or hardware in a mobile terminal.

[0080] like Figure 4 As shown, the question-answer quality judgment method includes the following steps:

[0081] S310: Obtain the question-answer pair text to be judged, and generate a question-answer pair quality judgment model based on the model generation method described in any embodiment of the present disclosure.

[0082] Specifically, the question-answer pair sample text may be question data and answer data in question-answering, knowledge sharing, etc. platforms. The question-answer pair quality judgment model generated based on the model generation method in any of the above embodiments can judge whether a question-answer pair is a question-answer pair that does not answer the question.

[0083] The model for judging the quality of question-answer pairs analyzes and judges the question and answer text as a whole, and can capture the effective lexical dependency between the question and the answer, and make a better judgment on whether the answer is irrelevant in complex situations. For example, for the question-answer pair "Question: What are the benefits of black rice? Answer: Purple rice is a variety of rice, belonging to the glutinous rice category, divided into purple japonica and purple glutinous rice. Generally speaking, purple glutinous rice is black glutinous rice; while black rice is a processed product of black rice, belonging to the japonica rice category, with two types of grains, indica and japonica, and black rice is non-glutinous rice", this technology can judge that the answer does not match the "benefit" and give a judgment that the answer is irrelevant.

[0084] Moreover, the question-answer quality judgment model is generated based on the training of the pre-trained language model. It uses the vocabulary of the pre-trained language model, has rich learnable predictions, can better capture the semantic features of the text, and can obtain the help of high-frequency co-occurring world knowledge to assist in judging whether the question-answer pair is irrelevant. For example, the model can learn "nutritional ingredients" ~ "protein, DHA"; "milk powder" ~ "infant", and thus successfully judge that "Question: What nutrients are there in milk powder? Answer: Protein: Whey protein has neurotransmitters that regulate sleep, helps infants sleep, and promotes infants' brain development..." This question-answer pair is not irrelevant.

[0085] In addition, during the training process of the question-answering quality judgment model, the multi-instance learning technology can reduce the impact of the training data set noise on the model, thereby improving the stability of the model and making the model's output results more confident.

[0086] S320: Input the question-answer pair text to be judged into the question-answer pair quality judgment model to obtain a quality judgment result of the question-answer pair to be judged.

[0087] By inputting the question-answer pair text to be judged whether it is a "irrelevant answer" into the question-answer pair quality judgment model, a judgment result can be obtained, that is, whether the question-answer pair is "irrelevant" or "answers the question".

[0088] The technical solution of the disclosed embodiment obtains a judgment result of the question-answer pair to be judged by inputting the question-answer pair to be judged into the question-answer quality judgment model obtained based on the model generation method in the above embodiment, and the accuracy confidence of the result is high. The problem of low accuracy of question-answer quality for judging complex question-answer pairs in the prior art is solved, and the judgment of question-answer quality is made when the question-answer content is complex, thereby improving the accuracy of the judgment of question-answer quality.

[0089] Embodiment 4

[0090] Figure 5 A structural schematic diagram of a model generation device provided in Embodiment 4 of the present disclosure is shown. Embodiments of the present disclosure are applicable to situations in which the model generation method provided in the above embodiments can be implemented through the question and answer quality judgment device provided in the present disclosure.

[0091] As shown in the figure, the model generation device in the embodiment of the present disclosure includes: a first sample analysis module 410 , a second sample analysis module 420 , a third sample analysis module 430 and a model generation module 440 .

[0092] Among them, the first sample analysis module 410 is used to obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector; the second sample analysis module 420 is used to input the word vector into the pre-trained language model to obtain a representation vector for each question-answer pair sample; the third sample analysis module 430 determines the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculates the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability; the model generation module 440 is used to generate a question-answer pair quality judgment model when the loss function result meets the preset conditions.

[0093] The technical solution of the disclosed embodiment uses a set containing multiple question-answer pair samples as a sample input unit, performs semantic analysis on the question-answer pair samples in the set, and then calculates the probability value of the predicted label of each question-answer pair in a set being the true label, and selects the question-answer pair with the highest probability value as the preferred model training data, adjusts the parameters of the model, and generates a question-answer pair quality judgment model when the loss function result meets the preset conditions. The problem of low accuracy in judging the question-answer quality of complex question-answer pairs in the prior art is solved, and the stability of the generated question-answer pair quality judgment model is enhanced, so that the question-answer pair quality judgment model makes higher accuracy judgments when the question-answer content is complex.

[0094] Optionally, the first sample analysis module is specifically used to: obtain question-answer pair texts with the same labels; divide the question-answer pair texts with the same labels into a preset number of question-answer pair sample sets, wherein the labels include labels that are not the answer to the question and labels that are the answer to the question.

[0095] Optionally, the first sample analysis module is further used to: match the question-answer pair samples with the vocabulary of the pre-trained language model to obtain word vectors corresponding to the sample data in each question-answer pair sample set.

[0096] Optionally, the third sample analysis module is specifically used to: input each of the representation vectors into a fully connected network to obtain the likelihood probability that the predicted label of each corresponding question-answer sample data pair is a correct classification label.

[0097] Optionally, the third sample analysis module realizes corresponding functions through a multi-instance learning mechanism in the model generation process.

[0098] Optionally, the second sample analysis module implements corresponding functions through a pre-trained language model encoding layer of the question-answer pair quality judgment model.

[0099] The model generation device provided in the embodiment of the present disclosure belongs to the same inventive concept as the model generation method provided in the above embodiment. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.

[0100] Embodiment 5

[0101] Figure 6 A structural schematic diagram of a question and answer quality judgment device provided in Embodiment 5 of the present disclosure is shown. The embodiment of the present disclosure can be applied to the situation where a question and answer pair is judged as to whether the answer is irrelevant to the question. The question and answer quality judgment method provided in the above embodiment can be implemented through the question and answer quality judgment device provided in the present disclosure.

[0102] like Figure 6 As shown, the question and answer quality judgment device in the embodiment of the present disclosure includes: a text and model acquisition module 510 and a question and answer quality judgment module 520.

[0103] Among them, the text and model acquisition module 510 is used to obtain the text of the question and answer pair to be judged, and generate a question and answer pair quality judgment model based on the model generation method described in any embodiment of the present disclosure; the question and answer quality judgment module 520 is used to input the question and answer pair text to be judged into the question and answer pair quality judgment model to obtain the quality judgment result of the question and answer pair to be judged.

[0104] The technical solution of the disclosed embodiment obtains a judgment result of the question-answer pair to be judged by inputting the question-answer pair to be judged into the question-answer quality judgment model obtained based on the model generation method in the above embodiment, and the accuracy confidence of the result is high. The problem of low accuracy of question-answer quality for judging complex question-answer pairs in the prior art is solved, and the judgment of question-answer quality is made when the question-answer content is complex, thereby improving the accuracy of the judgment of question-answer quality.

[0105] The question and answer quality judgment device provided in the embodiment of the present disclosure belongs to the same inventive concept as the question and answer quality judgment method provided in the above embodiment. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.

[0106] Embodiment 6

[0107] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 7 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 606 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] Typically, the following devices may be connected to the I / O interface 605: an input device 604 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 606 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 606, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0114] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a preset number of question-answer pair sample sets, and analyzes the sample data in each question-answer pair sample set to obtain a word vector; inputs the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample; determines the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculates the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability; generates a question-answer pair quality judgment model when the loss function result meets the preset conditions.

[0115] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device:

[0116] Obtain the text of the question and answer pair to be judged, and generate a question and answer pair quality judgment model based on the model generation method described in any embodiment of the present disclosure; input the question and answer pair text to be judged into the question and answer pair quality judgment model to obtain the quality judgment result of the question and answer pair to be judged.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0120] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] According to one or more embodiments of the present disclosure, [Example 1] provides a model generation method including:

[0123] Obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector;

[0124] Input the word vector into the pre-trained language model to obtain a representation vector for each question-answer pair sample;

[0125] Determine the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculate the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability;

[0126] When the loss function result meets the preset conditions, a question-answer pair quality judgment model is generated.

[0127] According to one or more embodiments of the present disclosure, [Example 2] provides the method of Example 1, further comprising:

[0128] The step of obtaining a preset number of question-answer pair sample sets includes:

[0129] Get the question-answer texts with the same labels;

[0130] The question-answer pair texts with the same labels are divided into a preset number of question-answer pair sample sets, wherein the labels include labels that are not answers to the question and labels that are answers to the question.

[0131] According to one or more embodiments of the present disclosure, [Example 3] provides the method of Example 1, further comprising:

[0132] The word vector is obtained by analyzing the sample data in each question-answer pair sample set, including:

[0133] Performing semantic word segmentation on the sample data to obtain a word segmentation result;

[0134] The word segmentation result is matched with the vocabulary of the pre-trained language model to obtain the word vector corresponding to the sample data in each question-answer pair sample set.

[0135] According to one or more embodiments of the present disclosure, [Example 4] provides the method of Example 3, further comprising:

[0136] The determining, based on the representation vector, the likelihood probability that the predicted label of each sample data pair is a correct classification label includes:

[0137] Each of the representation vectors is input into a fully connected network to obtain the likelihood probability that the predicted label of each corresponding question-answer sample data pair is the correct classification label.

[0138] According to one or more embodiments of the present disclosure, [Example 5] provides the method of Example 1, further comprising:

[0139] The method of determining the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculating the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability, is achieved through a multi-instance learning mechanism in the model generation process.

[0140] According to one or more embodiments of the present disclosure, [Example 6] provides the method of Example 1, further comprising:

[0141] The step of inputting the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample is achieved through the pre-trained language model encoding layer of the question-answer pair quality judgment model.

[0142] According to one or more embodiments of the present disclosure, [Example 7] provides a method for judging the quality of a question and answer, including:

[0143] Obtain the question-answer pair text to be judged, and generate a question-answer pair quality judgment model based on the model generation method described in any example;

[0144] The question-answer pair text to be judged is input into the question-answer pair quality judgment model to obtain the quality judgment result of the question-answer pair to be judged.

[0145] According to one or more embodiments of the present disclosure, [Example 8] provides a model generation device, including:

[0146] A first sample analysis module is used to obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector;

[0147] A second sample analysis module is used to input the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample;

[0148] A third sample analysis module determines the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculates the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability;

[0149] The model generation module is used to generate a question-answer pair quality judgment model when the loss function result meets the preset conditions.

[0150] According to one or more embodiments of the present disclosure, [Example 9] provides the device of Example 8, further comprising:

[0151] The first sample analysis module is used to:

[0152] Get the question-answer texts with the same labels;

[0153] The question-answer pair texts with the same labels are divided into a preset number of question-answer pair sample sets, wherein the labels include labels that are not answers to the question and labels that are answers to the question.

[0154] According to one or more embodiments of the present disclosure, [Example 10] provides the apparatus of Example 8, further comprising:

[0155] The first sample analysis module is further used for:

[0156] Performing semantic word segmentation on the sample data to obtain a word segmentation result;

[0157] The word segmentation result is matched with the vocabulary of the pre-trained language model to obtain the word vector corresponding to the sample data in each question-answer pair sample set.

[0158] According to one or more embodiments of the present disclosure, [Example 11] provides the device of Example 10, further comprising:

[0159] The third sample analysis module is used to:

[0160] Each of the representation vectors is input into a fully connected network to obtain the likelihood probability that the predicted label of each corresponding question-answer sample data pair is the correct classification label.

[0161] According to one or more embodiments of the present disclosure, [Example 12] provides the device of Example 8, further comprising:

[0162] The method of determining the likelihood probability that the predicted label of each question-answer pair sample is the correct label based on the representation vector, and calculating the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probability, is achieved through a multi-instance learning mechanism in the model generation process.

[0163] According to one or more embodiments of the present disclosure, [Example 13] provides the apparatus of Example 8, further comprising:

[0164] The step of inputting the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample is achieved through the pre-trained language model encoding layer of the question-answer pair quality judgment model.

[0165] According to one or more embodiments of the present disclosure, [Example 14] provides a question and answer quality judgment device, further comprising:

[0166] A text and model acquisition module, used to acquire the question-answer pair text to be judged, and generate a question-answer pair quality judgment model based on the model generation method described in any one of claims 1 to 6;

[0167] The question and answer quality judgment module is used to input the question and answer pair text to be judged into the question and answer pair quality judgment model to obtain the quality judgment result of the question and answer pair to be judged.

[0168] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A model generation method, characterized in that: include: Obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector; Inputting the word vector into a pre-trained language model with the question-answer pair sample set as a unit to obtain a representation vector for each question-answer pair sample, wherein the labels of the question-answer pair samples in a single question-answer pair sample set are the same; Determine the likelihood probability that the predicted label of each question-answer pair sample in the corresponding question-answer pair sample set is the correct label based on the representation vector of each question-answer pair sample in each question-answer pair sample set through the multi-instance learning mechanism in the model generation process, sort the likelihood probabilities to obtain the maximum value in the unique likelihood probabilities, and calculate the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probabilities; When the loss function result meets the preset conditions, a question-answer pair quality judgment model is generated.

2. The method according to claim 1, characterized in that: The step of obtaining a preset number of question-answer pair sample sets includes: Get the question-answer texts with the same labels; The question-answer pair texts with the same labels are divided into a preset number of question-answer pair sample sets, wherein the labels include labels that are not answers to the question and labels that are answers to the question.

3. The method according to claim 1, characterized in that The word vector is obtained by analyzing the sample data in each question-answer pair sample set, including: Performing semantic word segmentation on the sample data to obtain a word segmentation result; The word segmentation result is matched with the vocabulary of the pre-trained language model to obtain the word vector corresponding to the sample data in each question-answer pair sample set.

4. The method according to claim 3, characterized in that The determining, based on the representation vector, the likelihood probability that the predicted label of each sample data pair is a correct classification label includes: Each of the representation vectors is input into a fully connected network to obtain the likelihood probability that the predicted label of each corresponding question-answer sample data pair is the correct classification label.

5. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the word vector into a pre-trained language model to obtain a representation vector for each question-answer pair sample is achieved through the pre-trained language model encoding layer of the question-answer pair quality judgment model.

6. A method for judging the quality of a question and answer, characterized in that: include: Obtaining a question-answer pair text to be judged, and generating a question-answer pair quality judgment model based on the model generation method described in any one of claims 1 to 5; The question-answer pair text to be judged is input into the question-answer pair quality judgment model to obtain the quality judgment result of the question-answer pair to be judged.

7. A model generation device, characterized in that: include: A first sample analysis module is used to obtain a preset number of question-answer pair sample sets, and analyze the sample data in each question-answer pair sample set to obtain a word vector; A second sample analysis module is used to input the word vector into the pre-trained language model based on the question-answer pair sample set as a unit to obtain a representation vector for each question-answer pair sample, wherein the labels of the question-answer pair samples in a single question-answer pair sample set are the same; The third sample analysis module determines the likelihood probability that the predicted label of each question-answer pair sample in the corresponding question-answer pair sample set is the correct label based on the representation vector of each question-answer pair sample in each question-answer pair sample set through the multi-instance learning mechanism in the model generation process, sorts the likelihood probabilities to obtain the maximum value in the unique likelihood probabilities, and calculates the loss function according to the predicted label of the question-answer pair sample corresponding to the maximum value in the likelihood probabilities; The model generation module is used to generate a question-answer pair quality judgment model when the loss function result meets the preset conditions.

8. A question and answer quality judgment device, characterized in that: include: A text and model acquisition module, used to acquire the question-answer pair text to be judged, and generate a question-answer pair quality judgment model based on the model generation method described in any one of claims 1 to 5; The question and answer quality judgment module is used to input the question and answer pair text to be judged into the question and answer pair quality judgment model to obtain the quality judgment result of the question and answer pair to be judged.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the model generation method as described in any one of claims 1 to 5, or the question and answer quality judgment method as described in claim 6.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model generation method as described in any one of claims 1 to 5, or the question and answer quality judgment method as described in claim 6 is implemented.

Images