Text generation method and device, computer equipment and program product

Abstract

The present specification provides a text generation method. The method comprises the following steps: firstly, obtaining a question to be answered and a retrieved document set for the question to be answered; and inputting each document in the document set into the question and answer model to obtain a prediction result of the question and answer model for each document. And determining a first relevancy between the document and the question to be answered based on the prediction result corresponding to each document. And determining a target document input into the question and answer model according to the first relevancy, so as to generate a text for the to-be-answered question through the question and answer model. The target document input into the question and answer model is selected from the documents in the document set according to the prediction result generated by the question and answer model for each document, so that the target document input into the question and answer model better meets the requirements of the question and answer model, and the accuracy of the generated text is improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer application technology, and in particular, to a text generation method, apparatus, computer device, and program product. Background Art

[0002] The open-domain question-answering system aims to design a question-answering model to answer open questions that do not limit the scope of the subject. The open-domain question-answering system generally adopts a "retrieve-then-read" framework, under which the open-domain question-answering system mainly includes a retriever and a question-answering model. The specific workflow is that after obtaining the question to be answered, the retriever retrieves a preset number of documents related to the input question from a large corpus, and the question-answering model answers the question based on the documents retrieved by the retriever.

[0003] In open-domain question answering systems, there is a problem that the text generated by the question answering system is inaccurate. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a text generation method, apparatus, computer device, and program product.

[0005] According to a first aspect of one or more embodiments of this specification, a text generation method is proposed, the method comprising:

[0006] Obtaining questions to be answered and a document set retrieved for the questions to be answered;

[0007] For each document in the document set, input the document into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered;

[0008] Determine a first relevance between each document in the document set and the question to be answered according to the prediction result;

[0009] According to the first relevance corresponding to each document in the document set, a target document input into the question-answering model is determined, so that the question-answering model generates a text for the question to be answered according to the target document.

[0010] According to a second aspect of one or more embodiments of this specification, a text generation device is provided, the device comprising:

[0011] An acquisition module, used to acquire questions to be answered and a document set retrieved for the questions to be answered;

[0012] A first generating module is used to input each document in the document set into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered;

[0013] A determination module, configured to determine a first relevance between each document in the document set and the question to be answered according to the prediction result;

[0014] The second generating module is used to determine the target document to be input into the question-answering model according to the first relevance corresponding to each document in the document set, so that the question-answering model generates the text for the question to be answered according to the target document.

[0015] According to a third aspect of the embodiments of this specification, a computer program product is provided. When the computer program product is executed by a processor, the text generation method as described in the first aspect of the embodiments of this specification is implemented.

[0016] According to a fourth aspect of an embodiment of this specification, a computer device is provided, the computer device comprising:

[0017] processor;

[0018] a memory for storing processor-executable instructions;

[0019] The processor implements the text generation method as described in the first aspect of the embodiment of this specification by running the executable instructions.

[0020] This specification provides a text generation method. First, obtain a question to be answered and a document set retrieved for the question to be answered. Input each document in the document set into a question-and-answer model to obtain a prediction result of the question-and-answer model for each document. Determine a first relevance between the document and the question to be answered based on the prediction result corresponding to each document. Determine a target document to be input into the question-and-answer model based on the first relevance, so as to generate text for the question to be answered through the question-and-answer model. Select a target document to be input into the question-and-answer model for the document in the document set through the prediction result generated by the question-and-answer model for each document, so that the target document to be input into the question-and-answer model better meets the requirements of the question-and-answer model, thereby improving the accuracy of the generated text.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0023] Figure 1 It is a flowchart of a text generation method shown in this specification according to an exemplary embodiment.

[0024] Figure 2It is a schematic diagram of a text generation method shown in this specification according to an exemplary embodiment.

[0025] Figure 3 It is a clustering schematic diagram shown in this specification according to an exemplary embodiment.

[0026] Figure 4 It is a block diagram of a text generation device shown in this specification according to an exemplary embodiment.

[0027] Figure 5 It is a hardware structure diagram of a computer device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0029] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0030] Open-domain question-answering systems generally include: open-book and closed-book models. In the closed-book model, the question-answering model stores relevant domain knowledge in its own parameters and does not rely on an external knowledge base to provide background knowledge. The open-book model, on the other hand, calls an external knowledge base to obtain background knowledge for answering questions. The open-book model generally follows the principle of searching first and then reading. First, a preset number of relevant documents are retrieved from a large knowledge base such as a web page through a retriever, and then the retrieved documents are input into the question-answering model to obtain the answer text for the input question.

[0031] For the open-book model, there is a problem of low accuracy of the answer text.

[0032] The reason is that the retriever generally sorts the retrieved documents according to the similarity between each document and the question. However, due to memory limitations, the number of documents that the question-answering model can process at one time is limited. Accordingly, the question-answering model will take the top-ranked documents in the rearranged documents as input instead of all the retrieved documents.

[0033] When sorting, the retrieved documents are generally sorted based on the semantic relevance scores returned by the retriever. However, the level of semantic relevance is sometimes irrelevant to whether there is valuable information in the document. This makes the sorting results of the retriever sometimes not meet the needs of the question-answering model. For example, documents with low similarity rankings among the retrieved documents may contain valuable information for the question-answering model. However, due to the low ranking and the memory limitations of the question-answering model, these documents often cannot be input into the question-answering model. Similarly, documents with high rankings may not include valuable information. This makes it impossible for the question-answering model to fully utilize the retrieved documents, and even the documents input into the question-answering model may cause the question-answering model to output incorrect answer text.

[0034] It can be seen that due to the preference differences between the retriever and the question-answering model, the question-answering model may not be able to access and utilize the most helpful documents, which in turn makes the answer text output by the question-answering model inaccurate.

[0035] In order to solve the above problems, the related art generally adopts a re-ranking algorithm to calculate the semantic matching degree between the retrieved documents and the input questions, and re-ranks them based on the semantic matching degree. However, the re-ranking result often does not improve the accuracy of the answer of the question-answering model.

[0036] The reason is that the re-ranking algorithm is generally implemented based on a neural network, and the training of the re-ranking algorithm is generally independent of the question-answering model. This results in a preference difference between the re-ranking algorithm and the question-answering model, resulting in the retrieved documents containing valuable information still not being effectively used by the question-answering model, and even the top-ranked documents after re-ranking may still mislead the question-answering model to generate the wrong answer text.

[0037] Based on this, this specification provides a text generation method. First, obtain a question to be answered and a document set retrieved for the question to be answered. Input each document in the document set into a question-answering model to obtain a prediction result of the question-answering model for each document. Determine a first relevance between the document and the question to be answered based on the prediction result corresponding to each document. Determine a target document to be input into the question-answering model based on the first relevance, so as to generate text for the question to be answered through the question-answering model.

[0038] The prediction results generated by the question-answering model for each document are used to select the target documents for input into the question-answering model from the documents in the document set, so that the target documents for input into the question-answering model better meet the requirements of the question-answering model and improve the accuracy of the generated text.

[0039] Next, we will combine Figure 1 A text generation method shown in this specification is described. Figure 1 As shown, the text generation method includes the following steps:

[0040] Step 101: Obtain a question to be answered and a document set retrieved for the question to be answered.

[0041] Specifically, it is first necessary to obtain questions of the open domain question answering system, that is, questions to be answered, and obtain a document set retrieved by a retriever of the open domain question answering system.

[0042] In an optional implementation, the question to be answered may be a question input by a user. The document set may be a document set retrieved by a retriever obtained by the question-answering model in a large corpus. The document set may include a preset number of documents, each of which may include an article or a paragraph.

[0043] The retriever can be a retriever in the related art, and this specification does not limit the specific structure of the retriever. The retriever can retrieve a preset number of documents to form a document set, and in the subsequent steps, select some documents from the document set to input into the question-answering model.

[0044] Step 103: for each document in the document set, input the document into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered.

[0045] Step 105: Determine a first relevance between each document in the document set and the question to be answered according to the prediction result.

[0046] Specifically, there is a problem in the related art that the documents that are ranked high after sorting do not meet the requirements of the question-answering model. In order to solve this problem, each document is input into the question-answering model in this specification to obtain the prediction result of the question-answering model for the question corresponding to the document. Therefore, the first relevance corresponding to each document can be obtained based on the prediction result, and the first relevance is used to evaluate the practicality of the document for the question-answering model, and determine which documents contain useful information for the question-answering model. Therefore, documents that are more in line with the preferences of the question-answering model can be input into the question-answering model, making the results output by the question-answering model more accurate.

[0047] The prediction result may at least include: the answer to the question to be answered output by the question-answering model. Correspondingly, step 103 may specifically be to input the question to be answered into the question-answering model, and the question-answering model identifies and extracts the answer entity pointed to by each document in the document set.

[0048] The first relevance is an indicator for evaluating whether the document is relevant to the question to be answered. It can be used to evaluate whether the document is useful for the question-answering model to answer the question to be answered. In other words, it can be used to evaluate whether the document contains valuable information required for the question-answering model to answer the question to be answered.

[0049] Next, the specific implementation of steps 103 and 105 will be described through two specific examples.

[0050] In an optional implementation, the answer to the question to be answered may be a short text, such as a yes or no answer, a specific date, a specific number or a noun, etc.

[0051] Since the question-answering model generally generates one word each time when generating text, a text composed of multiple words is generated through multiple iterations. When generating each word, multiple alternative predicted answers and the predicted probability distribution corresponding to the alternative predicted answers are generated, and the sum of the predicted probabilities corresponding to the alternative answers is 1. Therefore, in the case where the answer is a short text, a single word can be used to represent the answer entity. The predicted probability distribution that the question-answering model needs to output can be used to determine the first relevance.

[0052] Correspondingly, the prediction result includes several predicted answers and the predicted probability distribution corresponding to each predicted answer, and the several predicted answers include no answer.

[0053] No answer means that the question-answering model considers the document irrelevant or unhelpful in answering the question to be answered. By setting no answer, you can filter documents. For example, if the predicted probability corresponding to no answer is the highest among multiple predicted answers, you can discard the document and not participate in the subsequent calculations. By setting no answer, you can also use the predicted probability corresponding to no answer to evaluate whether the document is useful to the question-answering model.

[0054] For example, Figure 2 As shown, the question to be answered is in which year did Company A start selling in Location C. There are multiple documents in the document set, which are sorted according to the relevance given by the retriever.

[0055] The question-answering model can extract multiple predicted answers from the first document. Among them, the predicted answer with the highest prediction probability is 1957, and the corresponding prediction probability is 0.814. The predicted probability of no answer is 0.0703. Correspondingly, the label of the document is 1957, that is, the answer extracted by the question-answering model from the document is 1957. This label can be used for calculations in the following text.

[0056] For the second document, although the retriever believes that the second document is semantically similar to the question to be answered, the answer with the highest predicted probability extracted from the document by the question-answering model is no answer, so the document can be ignored in subsequent processing.

[0057] The third and final document is similar to the first and will not be described again here.

[0058] In the case where the prediction result includes several predicted answers, the specific implementation method of step 105 may be: for each document in the document set, according to the first predicted probability that the document corresponds to no answer, determine the first relevance of the document to the question to be answered, and the first relevance is negatively correlated with the first predicted probability.

[0059] As for the specific implementation, in an optional implementation, the difference between 1 and the first predicted probability can be used as the first correlation. Figure 2 As shown, the first relevance of the first document is 1 minus the first predicted probability of no answer, which is 0.927. The calculation method of the first relevance of other documents is similar to that of the first document, and will not be repeated here.

[0060] In addition, the first correlation may be determined according to a preset monotonically decreasing function, where the function takes the first prediction probability as input and the first correlation as output.

[0061] The above two specific examples of determining the first correlation do not limit the present specification.

[0062] In the above implementation, the first relevance is determined based on the probability that the question-answering model cannot give an answer to the question to be answered from the document (i.e., the first predicted probability). The magnitude of the first relevance reflects the practicality and information content of the document. This facilitates the evaluation of whether each document has valuable information through the first relevance in the subsequent steps. In this way, useful documents can be determined based on the preference of the question-answering model, solving the problem of inaccurate output text caused by the preference difference between the retriever and the question-answering model.

[0063] In another optional implementation, the answer to the question to be answered can be a long text. For example, the question to be answered is how to achieve something, the specific meaning of a noun, etc. These questions cannot be represented by short and refined language, but can only be represented by long text.

[0064] In the above case, the prediction result can be the long text answer generated by the question-answering model after extracting information from the document for the question to be answered.

[0065] Correspondingly, the implementation method of step 105 may be to determine the similarity between the question to be answered and the prediction result, and use the similarity as the first correlation.

[0066] In this way, the question-answering model can be used to determine whether the document is practical for the question-answering model based on the answer given by the model, solving the problem of inaccurate output text caused by the preference difference between the retriever and the question-answering model.

[0067] Step 107, determining a target document to be input into the question-answering model according to the first relevance corresponding to each document in the document set, so that the question-answering model generates a text for the question to be answered according to the target document.

[0068] Specifically, after determining the first relevance, the target document can be determined according to the first relevance, and the document can be input into the question-answering model to obtain the text for the question to be answered, that is, the answer text. As described above, compared with the related art in which the document to be input into the question-answering model is determined based on the ranking results of the retriever or the re-ranking results of other algorithms, the method of this specification determines the target document based on the first relevance determined by the prediction results given by the question-answering model, so that the target document can better meet the requirements of the question-answering model and the generated answer text is more accurate.

[0069] The specific implementation of step 107, in an optional implementation, may be to reorder the documents in the document set according to the magnitude of the first relevance, and use the top k documents in the reordering result as target documents, input them into the question-answering model, and obtain the text for the question to be answered.

[0070] It should be noted that, although each document in the document set is input into the question-answering model in the above text, and the question-answering model generates the answer text of the question to be answered for each document, the prediction result in step 103 is not directly input into the question-answering model, or the prediction result in step 103 is not directly used as the result of the question-answering model, because when multiple documents are input into the question-answering model together, the question-answering model can give a more comprehensive answer based on multiple documents, which can improve the accuracy of the answer.

[0071] In addition, in addition to determining the target document based on the re-ranking result, step 107 may also cluster the documents and select multiple documents as target documents from the clusters that are most relevant to the question to be answered. In other words, step 107 may specifically include:

[0072] Step 1071 , clustering the documents in the document set according to the prediction result corresponding to each document in the document set to obtain a plurality of clusters.

[0073] Step 1072: Determine a second relevance between each cluster and the question to be answered based on the first relevance corresponding to each document.

[0074] Step 1073, selecting multiple documents from the cluster with the second highest relevance as input to the question-answering model.

[0075] In this way, the problem of inaccurate results of the question-answering model caused by other reasons can be further solved. Specifically, in the prior art, inaccurate answer text is not only caused by the inconsistency of preferences between the retriever and the question-answering model, but also affected by the document set containing inconsistent content. For example, the retrieved document set may include interfering information, or two documents may show contradictory information, and the contradictory information points to different answers to the question to be answered. For example, Figure 2 As shown, Figure 2 There is conflicting information in the first and third documents retrieved from .

[0076] When outputting answer text, the question-answering model generally needs to rely on the content of multiple documents to extract the correct answer, and is very sensitive to related content. Related documents containing interfering information will introduce noise and divert the attention of the question-answering model from the correct information, thus significantly reducing the accuracy of the question-answering model.

[0077] In the above implementation, documents are clustered based on the prediction results given by the question-answering model, so that documents pointing to the same or similar answers can be clustered together. When selecting target documents later, multiple documents are selected from the cluster most relevant to the question to be answered as target documents and input into the question-answering model, ensuring that the documents input into the question-answering model are documents with consistent contexts, avoiding the problem of contradictory information in the documents input into the question-answering model. This improves the accuracy of the answer text output by the question-answering model.

[0078] Among them, the specific implementation method of step 1071 can be: using the predicted answer of each document as the label corresponding to the document, wherein the predicted answer is determined according to the prediction result; clustering documents with the same label into a cluster to obtain multiple clusters.

[0079] For example, in the scenario of short text answers shown above, the predicted answer with the highest prediction probability can be used as the label, and documents with the same label can be clustered into one cluster.

[0080] In this way, for the question-answering model, documents with the same extracted answers can be clustered in one cluster, avoiding the problem of contradictory contents in multiple documents.

[0081] The specific clustering implementation method can be: take the answer with the highest prediction probability in the prediction results corresponding to each document as the label. Traverse the N documents in the document set. For each document, check whether its corresponding label overlaps with the label of the existing cluster. If there is overlap, add the retrieved document to the corresponding cluster; if there is no overlap, create a new cluster and use the label corresponding to the retrieved document as the label of the cluster.

[0082] For example, Figure 3 As shown, Figure 2 The result of document arrangement is as follows Figure 3 As shown, there are 8 documents remaining in the document set. The 8 documents are clustered according to their document labels, and documents with the same label are clustered into one cluster, resulting in 3 clusters.

[0083] In addition, the specific implementation method of step 1071 may also be: using a clustering algorithm to cluster the prediction results corresponding to each document in the document set to obtain multiple clusters.

[0084] For example, for long text answers, the answers extracted by the question-answering model from multiple documents are generally not exactly the same. Therefore, the features of the prediction results corresponding to each document can be extracted separately, and the clustering algorithm can be used to cluster the prediction results of multiple documents.

[0085] In this way, since the answers (i.e., prediction results) extracted by the question-answering model are clustered, it is also possible to avoid clustering documents with contradictory contexts in the same cluster, thereby avoiding the introduction of multiple documents with inconsistent contexts into the question-answering model.

[0086] A specific implementation of step 1072 may be to use, for each cluster, the average of the first relevances of the documents in the cluster as the second relevance corresponding to the cluster.

[0087] In this way, the second relevance can be determined based on the prediction results of the question-answering model, so that the second relevance can be used to judge whether each cluster is useful for the question-answering model to answer questions, thereby avoiding the problem of inconsistent preferences between the retriever and the question-answering model.

[0088] It should be noted that by determining the second relevance by the mean of the first relevance, it is possible to avoid the cluster containing the most documents having the highest second relevance. This can further make the target document more likely to be a document with a high first relevance, and further make the selected target document more useful for the question-answering model.

[0089] The specific implementation method of step 1072 may also be: sorting multiple documents in the document set according to the first relevance corresponding to each document, and determining the ranking of each document in the document set; determining the third relevance between each document and the question to be answered based on the ranking; the ranking is negatively correlated with the third relevance; for each cluster, determining the second relevance between the cluster and the question to be answered based on the third relevance between each document in the cluster and the question to be answered.

[0090] Since the first relevance between two documents with close rankings may be very close, for example, only differing by 0.01, by determining the third relevance based on the sorting ranking, the third relevance between different documents can be made more distinguishable, thereby making it easier to determine the second relevance corresponding to each cluster.

[0091] The third correlation can be determined by any method that ensures that the ranking is negatively correlated with the third correlation. The third correlation is described below by two specific embodiments, which are not intended to limit this specification.

[0092] Specifically, the method for determining the third relevance may be: inputting the ranking of the document into a preset relevance function to obtain the third relevance between the document and the question to be answered; wherein the preset similarity function is an exponential function or a piecewise function. The exponential function is a monotonically decreasing function; the function corresponding to each piece of the piecewise function is a constant function, and as the ranking increases, the constant value of the corresponding constant function decreases.

[0093] In addition, the preset correlation function may be any monotonically decreasing function, which is not limited in this specification.

[0094] Next, we will give specific examples of exponential functions and piecewise functions. The exponential function can be shown as formula (1):

[0095]

[0096] Among them, rel(rank di ) is the third relevance, rank di For ranking, d i Represents the document number. In the above function, the base e is greater than 1, and the independent variable ranking is on the denominator of the exponent. It can be seen that the exponential function shown in formula (1) is a monotonically decreasing function as the ranking increases.

[0097] The piecewise function can be specifically shown as formula (2):

[0098]

[0099] The meaning of each symbol is the same as that of formula (1). By using the piecewise function, the third relevance of each document can be divided according to the ranking interval.

[0100] A specific method for determining the second relevance may be: taking the sum or mean of the third relevance between each document in the cluster and the question to be answered as the second relevance between the cluster and the question to be answered.

[0101] The second relevance obtained by using the mean of the third relevance can be used to judge the average level of the third relevance of each document in each cluster. The relevance of the cluster is reflected by accumulating the third relevance scores of the retrieved documents in the cluster, and the summed second relevance can be used to judge the overall level of the third relevance of the documents in each cluster. The appropriate second relevance calculation method can be selected according to different needs.

[0102] In the case of summing, the calculation formula of the second correlation can be shown as formula (3):

[0103] score j =∑rel(rank di ),d i ∈C j (3)

[0104] Among them, score j represents the second correlation, j is the cluster number, C j represents the documents included in the jth cluster. The meanings of other symbols are the same as those in formula (1).

[0105] Through the second relevance, it is possible to evaluate which cluster's information is most useful for the question-answering model, and clustering can make the content of documents in a cluster more consistent, which in turn allows the question-answering model to receive a more consistent and reliable set of evidence, improving its ability to accurately identify and extract the correct answer.

[0106] like Figure 3 As shown, the third correlation is calculated according to the exponential function shown in formula (1), and the second correlation is calculated according to the function shown in formula (3), and the following can be obtained: Figure 3 Finally, it can be determined that cluster 1 is the second most relevant cluster and is the most useful cluster for the question-answering model.

[0107] The specific implementation of step 1073 may be: randomly selecting a number of documents from the cluster with the second highest relevance as target documents.

[0108] In addition, step 1073 may also be implemented by selecting the first k documents with the highest first relevance from the second cluster with the highest relevance as the input of the question-answering model, where k is a preset positive integer. k may be selected based on the performance of the question-answering model. By inputting the first k documents with the highest first relevance, the accuracy of the answer text may be improved.

[0109] In the above method, a method for re-ranking retrieved documents based on the question-answering model is proposed, which uses the predicted probability distribution of the question-answering model as a relevance metric, gives high priority to more useful and relevant documents from the perspective of the question-answering model, and filters out irrelevant documents. This solves the problem of preference differences between the retriever and the question-answering model.

[0110] To address the problem of inconsistent evidence faced by open domain question-answering systems, a retrieval document clustering method based on question-answering models is proposed. Documents pointing to similar answers are clustered, and documents with consistent contexts are selected from the clusters most relevant to the question and input into the question-answering model. Based on the needs of the question-answering model, evidence documents that are more helpful in answering questions and consistent with each other are given priority, reducing the interference caused by conflicting or contradictory information.

[0111] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus and a computer device to which it is applied.

[0112] like Figure 4 As shown, Figure 4 is a block diagram of a text generation device according to an exemplary embodiment of the present specification, the device comprising:

[0113] An acquisition module 410 is used to acquire questions to be answered and a document set retrieved for the questions to be answered;

[0114] A first generating module 420 is used to input each document in the document set into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered;

[0115] A determination module 430, configured to determine a first relevance between each document in the document set and the question to be answered according to the prediction result;

[0116] The second generating module 440 is used to determine the target document to be input into the question-answering model according to the first relevance corresponding to each document in the document set, so that the question-answering model generates a text for the question to be answered according to the target document.

[0117] In an optional implementation, the prediction result includes a number of predicted answers and a predicted probability distribution corresponding to each predicted answer, and the number of predicted answers includes no answer.

[0118] In an optional implementation, the determination module 430 is specifically used to determine, for each document in the document set, a first relevance of the document to the question to be answered based on a first predicted probability that the document has no answer, and the first relevance is negatively correlated with the first predicted probability.

[0119] In an optional implementation, the second generation module 440 includes:

[0120] A clustering submodule (not shown in the figure) is used to cluster the documents in the document set according to the prediction result corresponding to each document in the document set to obtain multiple clusters;

[0121] A determination submodule (not shown in the figure) is used to determine a second relevance between each cluster and the question to be answered based on the first relevance corresponding to each document;

[0122] A selection submodule (not shown in the figure) is used to select multiple documents from the cluster with the second highest relevance as input to the question-answering model.

[0123] In an optional implementation, the clustering submodule is specifically used to use the predicted answer of each document as the label corresponding to the document, wherein the predicted answer is determined according to the prediction result; and cluster documents with the same label into one cluster to obtain multiple clusters.

[0124] In an optional implementation, the determining submodule includes:

[0125] A ranking subunit (not shown in the figure), used to sort the multiple documents in the document set according to the first relevance corresponding to each document, and determine the ranking of each document in the document set;

[0126] A first determination subunit (not shown in the figure) is used to determine a third relevance between each document and the question to be answered based on the ranking; the ranking is negatively correlated with the third relevance;

[0127] The second determination subunit (not shown in the figure) is used to determine, for each cluster, the second relevance between the cluster and the question to be answered according to the third relevance between each document in the cluster and the question to be answered.

[0128] In an optional embodiment, the first determination subunit is specifically used to: input the ranking of the document into a preset relevance function to obtain a third relevance between it and the question to be answered; wherein the preset similarity function is an exponential function or a piecewise function; the exponential function is a monotonically decreasing function; the function corresponding to each segment of the piecewise function is a constant function, and as the ranking increases, the constant value of the corresponding constant function decreases.

[0129] In an optional implementation manner, the second determining subunit is specifically configured to: use the sum or mean of the third relevance between each document in the cluster and the question to be answered as the second relevance between the cluster and the question to be answered.

[0130] In an optional implementation, the selection submodule is specifically used to: select the first k documents with the highest first relevance from the cluster with the highest second relevance as inputs to the question-answering model, where k is a preset positive integer.

[0131] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0132] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0133] The present specification also provides a computer device, comprising at least: a processor; a memory for storing processor executable instructions; wherein the processor implements the above-mentioned text generation method by running the executable instructions.

[0134] For computer equipment specific examples, such as Figure 5 As shown, Figure 5 A hardware structure diagram of a computer device is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0135] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0136] The memory 1020 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020.

[0137] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0138] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0139] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0140] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0141] The embodiments of this specification also provide a computer program product, which implements the above-mentioned text generation method when executed by a processor.

[0142] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned text generation method is implemented.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0146] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A text generation method, the method comprising: Obtaining questions to be answered and a document set retrieved for the questions to be answered; For each document in the document set, input the document into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered; Determine a first relevance between each document in the document set and the question to be answered according to the prediction result; According to the first relevance corresponding to each document in the document set, a target document input into the question-answering model is determined, so that the question-answering model generates a text for the question to be answered according to the target document.

2. The method according to claim 1, wherein: The prediction result includes several predicted answers and the predicted probability distribution corresponding to each predicted answer, and the several predicted answers include no answer.

3. The method according to claim 2, wherein: Determining a first relevance between each document in the document set and the question to be answered according to the prediction result includes: For each document in the document set, a first relevance between the document and the question to be answered is determined according to a first predicted probability that the document has no answer, and the first relevance is negatively correlated with the first predicted probability.

4. The method according to claim 1, wherein: The step of determining the document to be input into the question-answering model according to the first relevance corresponding to each document in the document set includes: According to the prediction result corresponding to each document in the document set, the documents in the document set are clustered to obtain multiple clusters; Determining a second relevance between each cluster and the question to be answered according to the first relevance corresponding to each document; A plurality of documents are selected from the cluster with the second highest relevance as inputs of the question-answering model.

5. The method according to claim 4, wherein: According to the prediction result corresponding to each document in the document set, the documents in the document set are clustered to obtain multiple clusters, including: Using the predicted answer for each document as a label corresponding to the document, wherein the predicted answer is determined according to the prediction result; Documents with the same label are clustered into one cluster, resulting in multiple clusters.

6. The method according to claim 4, wherein: The determining, according to the first relevance corresponding to each document, a second relevance between each cluster and the question to be answered comprises: Sorting a plurality of documents in the document set according to a first relevance corresponding to each document, and determining a ranking of each document in the document set; Determining a third relevance between each document and the question to be answered based on the ranking; the ranking is negatively correlated with the third relevance; For each cluster, a second relevance between the cluster and the question to be answered is determined according to a third relevance between each document in the cluster and the question to be answered.

7. The method according to claim 6, wherein: The third relevance between the document and the question to be answered includes: Inputting the ranking of the document into a preset relevance function to obtain a third relevance between the document and the question to be answered; Among them, the preset similarity function is an exponential function or a piecewise function; the exponential function is a monotonically decreasing function; the function corresponding to each segment of the piecewise function is a constant function, and as the ranking increases, the constant value of the corresponding constant function decreases.

8. The method according to claim 6, wherein: Determining the second relevance between the cluster and the question to be answered according to the third relevance between each document in the cluster and the question to be answered includes: The sum or mean of the third relevance between each document in the cluster and the question to be answered is used as the second relevance between the cluster and the question to be answered.

9. The method according to claim 4, wherein: The selecting a plurality of documents from the cluster with the second highest relevance as inputs of the question-answering model includes: From the cluster with the second highest relevance, the first k documents with the first highest relevance are selected as inputs of the question-answering model, where k is a preset positive integer.

10. A text generation device, comprising: An acquisition module, used to acquire questions to be answered and a document set retrieved for the questions to be answered; A first generating module is used to input each document in the document set into the question-answering model to obtain a prediction result corresponding to the document for the question to be answered; A determination module, configured to determine a first relevance between each document in the document set and the question to be answered according to the prediction result; The second generating module is used to determine the target document to be input into the question-answering model according to the first relevance corresponding to each document in the document set, so that the question-answering model generates the text for the question to be answered according to the target document.

11. A computer program product, wherein: When the computer program product is executed by a processor, the method described in any one of claims 1 to 9 is implemented.

12. A computer device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

Images