Method and device for evaluating rationality of text and evaluation method and device

Abstract

The invention relates to a method for evaluating text rationality, which comprises the following steps: obtaining a first text and a second text, the second text is generated by a text generation system based on the first text, and the second text comprises at least one sub-text, each sub-text in the at least one sub-text has a text source identifier, and the text source identifier is a text source identifier; the text source identifier indicates a source text from which the corresponding sub-text is derived; obtaining at least one source text based on the data source identifier of the at least one sub-text; for each source text in the at least one source text, generating a summary text corresponding to the source text based on content related to a text topic in the source text, the text topic being determined based on the first text; for each sub-text in the at least one sub-text, determining the reasonability of the sub-text based on a summary text of the source text corresponding to the sub-text; and evaluating the rationality of the second text based on the rationality of each sub-text in the at least one sub-text.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing, and in particular, to a method and device for evaluating text rationality, an evaluation method and device for a text generation system, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Natural Language Processing (NLP) is an important field in the fields of computer science and artificial intelligence. It integrates multiple disciplines such as linguistics, computer science, and mathematics. Natural language processing mainly studies how to effectively communicate between people and computers through natural language, and can be applied to a variety of scenarios such as question answering, public opinion monitoring, automatic summarization, opinion extraction, and text semantic comparison. In these scenarios, many scenarios involve generating text with the help of natural language processing systems. How to evaluate the rationality of the generated text is a highly concerned issue in this field. Summary of the invention

[0003] In view of this, the present disclosure provides a method and apparatus for evaluating text rationality, an evaluation method and apparatus for a text generation system, a computing device, a computer-readable storage medium, and a computer program product, which help solve the above-mentioned problems.

[0004] According to one aspect of the present disclosure, a method for evaluating text rationality is provided, comprising: acquiring a first text and a second text, wherein the second text is generated by a text generation system based on the first text, and the second text includes at least one sub-text, each of the at least one sub-text has a text source identifier, and the text source identifier indicates a source text from which the corresponding sub-text originates; acquiring at least one source text based on a data source identifier of the at least one sub-text; for each of the at least one source text, generating a summary text corresponding to the source text based on content related to a text theme in the source text, wherein the text theme is determined based on the first text; for each of the at least one sub-text, determining the rationality of the sub-text based on a summary text of the source text corresponding to the sub-text; and evaluating the rationality of the second text based on the rationality of each of the at least one sub-text.

[0005] In some embodiments, the above method further includes: determining a text theme based on at least one keyword in the first text.

[0006] In some embodiments, generating a summary text corresponding to the source text based on the content related to the text topic in the source text includes: obtaining at least one text block of the source text, wherein the at least one text block has a preset arrangement order; initializing a current text block and a local summary text, wherein the current text block is initialized as the first text block in the at least one text block; iteratively performing the following steps: updating the local summary text based on the content related to the text topic in the current text block; in response to the current text block not being the last text block in the at least one text block, updating the current text block to the next text block according to the preset arrangement order; in response to the current text block being the last text block in the at least one text block, determining the summary text based on the local summary text.

[0007] In some embodiments, updating the local summary text based on the content related to the text topic in the current text block includes: summarizing the local summary text and the content related to the text topic in the current text block, and updating the local summary text based on the summary result.

[0008] In some embodiments, obtaining at least one text block of the source text includes: obtaining at least one text block based on at least one of the following items: a natural language segment in the source text, a structured parsing result of the source text.

[0009] In some embodiments, for each sub-text of at least one sub-text, based on the summary text of the source text corresponding to the sub-text, determining the rationality of the sub-text includes: for each sub-text of at least one sub-text, determining a classification result of the sub-text according to the degree of association between the sub-text and the summary text of the source text corresponding to the sub-text, the classification result indicating whether the sub-text is reasonable or unreasonable.

[0010] In some embodiments, evaluating the rationality of the second text based on the rationality of each subtext in at least one subtext includes: evaluating the rationality of the second text based on the proportion of subtexts classified as reasonable in the at least one subtext.

[0011] According to another aspect of the present disclosure, there is provided an evaluation method for a text generation system, comprising: obtaining multiple groups of texts, wherein each group of texts includes a first text and a second text, and the second text is generated by the text generation system based on the first text; for each group of texts in the multiple groups of texts, evaluating the rationality of the second text in each group of texts according to the method for evaluating text rationality described in various embodiments of the aforementioned aspects; and evaluating the rationality of the text generation system based on the rationality of at least a portion of the second text in the multiple groups of texts.

[0012] According to another aspect of the present disclosure, there is provided an apparatus for evaluating the rationality of a text, comprising: a first acquisition module configured to acquire a first text and a second text, wherein the second text is generated by a text generation system based on the first text, and the second text includes at least one sub-text, each of the at least one sub-text has a corresponding text source identifier, and the text source identifier indicates a source text from which the corresponding sub-text originates; a second acquisition module configured to acquire at least one source text based on a data source identifier of the at least one sub-text; a first intelligent agent configured to generate a summary text corresponding to each of the at least one source text based on content related to the text theme in the source text, wherein the text theme is determined based on the first text; a second intelligent agent configured to determine the rationality of each of the at least one sub-text based on the summary text of the source text corresponding to the sub-text; and an evaluation module configured to evaluate the rationality of the second text based on the rationality of each of the at least one sub-text.

[0013] In some embodiments, the above-mentioned device also includes: a third agent configured to: determine the text theme based on at least one keyword in the first text.

[0014] According to another aspect of the present disclosure, there is provided an evaluation device for a text generation system, comprising: an acquisition module, configured to acquire multiple groups of texts, wherein each group of texts comprises a first text and a second text, and the second text is generated by the text generation system based on the first text; a text evaluation module, configured to evaluate the rationality of the second text in each group of texts for each group of texts by using the device for evaluating the rationality of texts according to the aforementioned aspects; and a system evaluation module, configured to evaluate the rationality of the text generation system based on the rationality of at least a portion of the second text in the multiple groups of texts.

[0015] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute the method according to any embodiment of the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the method according to any embodiment of the aforementioned aspect is executed.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which implement the steps of the method described in any embodiment of the aforementioned aspect when executed by a processor.

[0018] Through the scheme for evaluating the rationality of text provided by the present disclosure, the rationality of the second text generated based on the first text can be evaluated, wherein a summary text can be obtained based on the source text on which the second text is based, based on the text theme reflected by the first text, and then the rationality of the second text can be evaluated based on the summary text. This evaluation method can more effectively evaluate whether there is content in the second text that is inconsistent with the text theme reflected by the first text, that is, evaluate whether the content in the second text is logically related to the text theme reflected by the first text, so as to effectively reflect the degree of logical relevance between the content of the second text and the text theme reflected by the first text, that is, reflect the logical rationality of the second text.

[0019] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 An example architecture diagram of a text generation system according to some embodiments of the present disclosure is schematically shown;

[0022] Figure 2 An example flow chart of a method for evaluating text rationality according to some embodiments of the present disclosure is schematically shown;

[0023] Figure 3 An exemplary architecture diagram of a system for evaluating text rationality according to some embodiments of the present disclosure is schematically shown;

[0024] Figure 4 An example flow chart of an evaluation method for a text generation system according to some embodiments of the present disclosure is schematically shown;

[0025] Figure 5 An example architecture diagram of an evaluation system for a text generation system according to some embodiments of the present disclosure is schematically shown;

[0026] Figure 6 An example block diagram of an apparatus for evaluating text rationality according to some embodiments of the present disclosure is schematically shown;

[0027] Figure 7 An example block diagram of an evaluation device for a text generation system according to some embodiments of the present disclosure is schematically shown;

[0028] Figure 8An example block diagram of a computing device according to some embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Figure 1 The exemplary architecture of the text generation system 100 according to some embodiments of the present disclosure is schematically shown. Figure 1 As shown, the text generation system 100 may include a model or model group 110 related to text generation, and include an interface for receiving input text. The input text may be, for example, a question, a keyword, or other text input by a user. The received input text may be provided to the model or model group 110, wherein, optionally, the input text may be provided to the model or model group 110 directly or after preprocessing. The model or model group 110 may generate an output text 140 based on the input text or the preprocessed input text. More specifically, in some embodiments, the model or model group 110 may retrieve relevant texts in a database 130 based on the input text or the preprocessed input text, and generate output texts based on the retrieved texts, such as by text concatenation, text summarization, or by other means. Exemplarily, the text in the database 130 may be vectorized and stored in advance. Subsequently, when the input text is received, the input text may be vectorized, and the vector of the input text may be matched with the text vector in the database for similarity, and the k results with the highest similarity are returned, wherein k is a pre-set integer greater than 0. Based on the text blocks corresponding to the k results, a concatenated text can be obtained by text concatenation, and then the concatenated text can be summarized by the large model to obtain the output text. Optionally, the above process can be implemented by RAG (Retrieval-Augmented Generation) technology or other technologies.

[0030] However, when recalling text blocks by vector similarity, since vector similarity can reflect the degree of semantic similarity but is difficult to accurately represent logical relationships, it is possible to recall unreasonable text blocks. For example, although the recalled text blocks have semantic similarities with the input text, the logical subjects and themes of the two may be different. For example, assuming that the input text indicates to analyze the sales changes of product B of brand A, at this time, when recalling text blocks based on vector similarity, one or more text blocks related to the sales changes of product B of brand A can be obtained, but at the same time, some text blocks irrelevant to it may also be obtained. These irrelevant text blocks may not explicitly contain the name of the brand or product, but are actually related to other brands or other products (for example, the article to which the irrelevant text block belongs may be about other brands or other products, or other brands or other products may be mentioned in other text blocks before the irrelevant text block), for example, related to the sales changes of product B of other brands, related to the sales changes of other products of brand A, or related to the sales changes of other products of other brands. Obviously, these irrelevant text blocks will have a negative impact on the rationality of the final output text.

[0031] In addition, due to limitations in model performance, for example, some models may not be strong enough in following instructions and logical reasoning, and there may also be logical problems in the generated text.

[0032] In related technologies, the quality of generated text can be measured based on the text overlap rate and similarity between the input text and the output text, or the quality of generated text can be determined by scoring with a large model. However, neither of these two methods can truly reflect the logical relationship between the input text and the output text, and cannot accurately identify the problem of irrelevant text blocks in the above example.

[0033] Based on the above considerations, the present disclosure proposes a new solution for evaluating text rationality.

[0034] Figure 2 The following schematically shows an example flow chart of a method 200 for evaluating text rationality according to some embodiments of the present disclosure. The method 200 may be performed by any suitable device or system. For example, the method 200 may be implemented by one or more algorithm models and / or applications deployed on one or more computing devices. Figure 2 As shown, method 200 may include steps 210 to 250, which are specifically as follows.

[0035] In step 210, a first text and a second text may be obtained, wherein the second text may be generated by a text generation system based on the first text. The second text may include at least one sub-text, and each of the at least one sub-text may have a text source identifier, which may indicate the source text from which the corresponding sub-text originates. In some examples, the first text may be input text provided by a user to the text generation system, such as questions or instructions input by the user, and the second text may be an answer to the user's question or a text that complies with the user's instructions, etc., generated by the text generation system. Optionally, the first text and the second text may be read from an internal or external database, received from the text generation system, or obtained from user input. Exemplarily, the text generation system may be associated with a reference text. Figure 1 Similar to the system described, it can search in one or more designated databases based on the first text, and generate a second text corresponding to the first text based on the search results. For example, the second text can be generated based on multiple sub-texts related to the first text retrieved through text splicing and other steps. Optionally, the text source identifier of each sub-text can be an index, title or other form of identifier of the corresponding source text, so that based on the identifier, the corresponding source text can be uniquely determined. In the present disclosure, a sub-text is derived from a source text, which should be understood as: the sub-text is a part of the source text, or the sub-text is obtained based on part of the content of the source text, such as by summarizing, rewriting or other processing of part of the content of the source text.

[0036] At step 220, at least one source text may be obtained based on the data source identifier of at least one subtext. Exemplarily, the corresponding source texts may be read in sequence in the corresponding database based on the data source identifiers corresponding to the respective subtexts, wherein the respective subtexts in the at least one subtext may all be derived from different source texts, or some of the subtexts may be derived from the same source text.

[0037] In step 230, for each source text in at least one source text, a summary text corresponding to the source text can be generated based on the content related to the text theme in the source text, wherein the text theme is determined based on the first text. Exemplarily, the source text can be processed based on the text theme, and the content irrelevant to the text theme is filtered out, and only the content related to the text theme is summarized to obtain the corresponding summary text. Optionally, this step can be performed by a pre-trained model or based on a preset algorithm rule. For example, the model can be pre-trained to output a summary text around the expected text theme based on the input text, wherein the model can take various appropriate structures, such as the structure of various large language models.

[0038] Optionally, depending on the specific application scenario, the text theme can be the first text itself, or it can also be obtained by summarizing or extracting keywords from the first text. For example, in some embodiments, before step 230, method 200 may further include: determining the text theme based on at least one keyword in the first text. The keywords in the first text can reflect the key information therein, so it is helpful to accurately and efficiently determine the text theme. Optionally, this step can be performed by a pre-trained model or based on a preset algorithm rule. For example, the model can be pre-trained to output the text theme of the input text based on the input text, wherein the model can adopt various appropriate structures, such as the structure of various large language models. Exemplarily, the keywords in the first text can include the name of the subject involved (such as a person, event, brand, product, object, etc.), and can also include other words related to the theme of the first text. In addition, the determination process of this text theme can also be set and optimized for specific scenarios, such as optimizing the corresponding model or algorithm for certain target scenarios to improve the adaptability to the corresponding scenarios.

[0039] In step 240, for each subtext in at least one subtext, the rationality of the subtext can be determined based on the summary text of the source text corresponding to the subtext. Optionally, this step can be performed by a pre-trained model or based on a preset algorithm rule. For example, the model can be pre-trained to output text rationality based on two input texts, wherein the model can adopt various appropriate structures, such as the structure of various large language models. Exemplarily, the rationality of the subtext can be determined by the degree of association between the subtext and the summary text of the corresponding source text, wherein the degree of association can, for example, measure the degree of information overlap or similarity between the subtext and the corresponding summary text, whether the subtext can be included or derived by the corresponding summary text, etc. A higher degree of association between the subtext and the summary text of the corresponding source text can correspond to a higher rationality of the subtext. Optionally, the rationality can be represented by scores, grades, classification results (such as binary classification), etc.

[0040] In step 250, the rationality of the second text may be evaluated based on the rationality of each subtext in at least one subtext. Optionally, the rationality of the second text may be evaluated by statistically summarizing the rationality of each subtext. For example, the rationality of each subtext may be statistically summarized by calculating the average value, median value, etc. of the rationality of each subtext, or by counting the number, proportion, etc. of subtexts at different rationality levels.

[0041] Through method 200, for each sub-text in the second text generated based on the first text, the logical relevance of the sub-text to the text theme of the first text can be determined by means of the context of the corresponding source text, thereby determining its rationality. Furthermore, the rationality of the second text can be evaluated based on the rationality of each sub-text. In this way, the logical relevance of the sub-texts contained in the second text to the first text can be effectively taken into consideration during the evaluation process, thereby effectively evaluating the logical relationship between the second text and the first text.

[0042] In some embodiments, the aforementioned step 230 may include: for each source text in at least one source text, obtaining at least one text block of the source text, wherein at least one text block has a preset arrangement order; initializing the current text block and the local summary text, wherein the current text block is initialized as the first text block in at least one text block; and performing an iteration step, based on the at least one text block, gradually updating the local summary text to determine the summary text corresponding to the source text. The iteration step may include: updating the local summary text based on the content related to the text theme in the previous text block; in response to the current text block not being the last text block in at least one text block, updating the current text block to the next text block according to the preset arrangement order; in response to the current text block being the last text block in at least one text block, determining the summary text based on the local summary text. Exemplarily, when updating the local summary text based on the content related to the text theme in the current text block, the local summary text and the content related to the text theme in the current text block may be summarized, and the local summary text may be updated based on the summary result. When the current text block is irrelevant to the text topic, the logical relationship between the current text block and the previous text block can be taken into account when updating the local summary text, rather than summarizing the current text block in isolation. This helps to more accurately reflect the correct logic of the content in the source text related to the desired text topic.

[0043] Through the iterative process in the above embodiment, starting from the first text block of the source text, the local summary text can be updated according to the next text block in sequence around the text theme. In this way, the contextual association between different text blocks in the source text can be fully considered, and the correct logical relationship therein can be sorted out, which helps to improve the accuracy of the evaluation results.

[0044] Exemplarily, at least one text block of the source text may refer to at least one paragraph in the source text. And, exemplarily, the order of appearance of at least one text block in the source text may be regarded as its preset arrangement order. Exemplarily, at least one text block of the source text may be obtained based on at least one of the following: a natural paragraph in the source text, a structured parsing result of the source text. A natural paragraph in the source text may refer to a sentence or sentence group, a natural paragraph or a chapter in the source text, etc. The structured parsing result of the source text may refer to the directory structure or chapter structure of the source text, or a text structure parsed based on other methods. This structured parsing result helps to enhance the logical compliance effect in the summary process, and helps to improve the summary efficiency. Exemplarily, through the titles in the directory or chapters, it may be known that some text parts in the source text are completely irrelevant to the text theme of concern, so these text parts can be directly skipped.

[0045] In some embodiments, the aforementioned step 240 may include: for each sub-text in at least one sub-text, according to the degree of association between the sub-text and the summary text of the source text corresponding to the sub-text, determining the classification result of the sub-text, the classification result indicating whether the sub-text is reasonable or unreasonable. Exemplarily, the sub-texts can be classified with the help of a pre-trained model. Further exemplarily, the degree of association between the sub-text and the summary text of the corresponding source text can be scored, and the sub-text can be classified as reasonable or unreasonable based on the comparison between the score and a preset threshold, wherein the preset threshold can be determined according to the training situation during the model training process. Alternatively, the above-mentioned two-classification process can also be implemented in other ways. By classifying the sub-texts into two categories of reasonable or unreasonable, the rationality of each sub-text can be more concisely characterized, which helps to simplify the subsequent evaluation steps and improve the evaluation efficiency.

[0046] In such embodiments, the aforementioned step 250 may include: based on the proportion of subtexts with reasonable classification results in at least one subtext, evaluating the rationality of the second text. Exemplarily, assuming that the second text includes 4 subtexts, of which the classification results of 3 subtexts are reasonable and the classification result of 1 subtext is unreasonable, the proportion of reasonable subtexts in the 4 subtexts is 75%. Optionally, the rationality of the second text can be determined as the proportion of subtexts with reasonable classification results in at least one subtext itself, for example, in the above example, it is determined to be 75%, or it can also be determined as other quantities related to the proportion, such as converting the proportion into a corresponding rationality level according to a preset rule, converting the proportion into a binary classification result according to a preset threshold, etc. Evaluating the rationality of the second text according to the proportion of subtexts with reasonable classification results in the second text helps to reduce the calculation or processing complexity of the evaluation process and improve the evaluation efficiency.

[0047] Figure 3 An exemplary architecture diagram of a system 300 for evaluating text rationality according to some embodiments of the present disclosure is schematically shown. Figure 3 As shown, system 300 may include agent 310, agent 320, and agent 330. Agent 310 may be regarded as a topic agent, which may receive a first text, such as a question or instruction raised by a user, and determine a text topic based on the first text. Agent 320 may be regarded as a summarize agent, which may receive one or more source texts, and summarize each of the one or more source texts around the text topic to obtain a summary text corresponding to each source text, wherein the one or more source texts are obtained based on the data source identifier of the subtext of the second text. Agent 330 may be regarded as a critic agent, which may judge the rationality of each subtext based on each subtext and the corresponding summary text. Furthermore, the rationality of the second text may be evaluated based on the rationality of each subtext. In some application scenarios, the first text itself may be regarded as a text topic, and in this case, agent 310 may not be required.

[0048] In the present disclosure, an agent may be understood as an intelligent entity that may have computing and reasoning capabilities, and may have the ability to exchange information with other agents or other systems. Exemplarily, each of the above-mentioned agents 310, 320, and 330 may include an AI (Artificial Intelligence) model, such as a large language model (LLM). Further exemplarily, based on a pre-trained LLM, it may be guided and set through prompts (prompt input) to focus more on specific information and improve its performance on specific tasks, thereby better implementing the various functions described in the various embodiments above.

[0049] For ease of understanding, the process of evaluating the rationality of text described in at least some of the above embodiments will be introduced below with a specific example. For example, the first text may be "Development trend of brand A in 2023", and the second text may include the following content:

[0050] “According to the report analysis, the development trends of Brand A in 2023 are as follows:

[0051] 1. Sales growth: Brand A's sales continue to maintain strong growth momentum, with global sales reaching xx in 2022, a year-on-year increase of xx. [1]

[0052] With the announcement of price cuts for Model Y in China, North America and other regions, Brand A sales are expected to be further boosted. [1]

[0053] 2. Increased value of each vehicle: Since the company has developed forging and extrusion technologies, the value of each vehicle provided by the company to Brand A has increased. [1]

[0054] Therefore, the value of Brand A to the company’s bicycle accessories is also expected to increase. [1]

[0055] 3. Global expansion: Brand A plans to add more factories around the world in the next few years to meet market demand. [2]

[0056] This will bring more business opportunities to the company.[3]

[0057] 4. Diversification: Brand A is expanding its product line to include solar roofs, energy storage systems, electric trucks, etc. [4]

[0058] Companies can also take this opportunity to expand their business scope.[5]

[0059] In summary, the development trend of Brand A in 2023 is positive, and it is expected to bring more business opportunities and growth momentum to the company. [3]”

[0060] In the above example, the second text includes 9 sub-texts. And, in this example, each sub-text is a sentence. It can be seen in the above example that there is an index at the end of each sub-text, such as the words "[1]", "[2]", "[3]", "[4]", "[5]" shown in the above second text. The index is used to indicate the corresponding source text, such as the corresponding article, research report, etc. After obtaining the second text, the corresponding source text can be obtained according to the index of each sub-text. In the above example, there are 5 source texts.

[0061] The first text can be entered into Figure 3The agent 310 shown in the figure can extract key information and generate a text theme. In this example, the generated text theme can be, for example, "Brand A Development Trend". The text theme can be provided to the agent 320. In addition, the above five source texts can also be provided to the agent 320 in sequence. For each source text, the agent 320 can combine the above text theme, start from the first paragraph of the source text, recursively call the large model to continuously summarize with the next paragraph, gradually sort out the correct logic of the source text and the global view of the source text, and finally obtain the corresponding summary text around the text theme. Optionally, these summary texts can be stored locally, stored in an external designated memory, or directly provided to the agent 330. The agent 330 can traverse each subtext in the second text, and, for each subtext, can obtain the corresponding summary text, and judge whether the subtext is logical based on the summary text, and then generate a binary classification judgment result, wherein the binary classification judgment result indicates whether the subtext is reasonable or unreasonable. After the traversal is completed, 9 binary classification judgment results corresponding to the above 9 sub-texts can be obtained. The overall rationality of the second text can be evaluated according to the proportion of reasonable sub-texts in the 9 sub-texts.

[0062] The various embodiments described above can be used in any scenario where the rationality of a second text generated based on a first text needs to be evaluated. In some embodiments, it can be used to evaluate a text generation system. Figure 4 Schematically shows an example flow chart of an evaluation method 400 for a text generation system according to some embodiments of the present disclosure. The method 400 may be performed by any suitable device or system. Figure 4 As shown, method 400 may include steps 410 to 430, which are specifically as follows.

[0063] In step 410, multiple groups of texts can be obtained, wherein each group of texts includes a first text and a second text, and the second text is generated by a text generation system based on the first text. Optionally, the multiple groups of texts can be read from an internal or external database, received from a text generation system, or obtained from user input. The number of multiple groups of texts can be set according to specific needs, and the present disclosure is not limited in this regard. Regarding the first text, the second text, and the text generation system, please refer to the relevant description of method 200 in the above text, which will not be repeated here.

[0064] In step 420, for each of the plurality of text groups, the rationality of the second text in each text group may be evaluated according to various embodiments of the method 200 described above. Step 420 may be performed according to various embodiments of the method 200 described above, and will not be described in detail herein.

[0065] In step 430, the rationality of the text generation system can be evaluated based on the rationality of at least a part of the second texts in the multiple groups of texts. Exemplarily, the rationality of the text generation system can be evaluated based on the average value of the rationality of at least a part of the second texts in the multiple groups of texts, or the rationality of the text generation system can be evaluated based on the median, mode, etc. of the rationality of each second text, wherein the average value, median, mode, etc. can be directly used as a rationality index of the text generation system, or can be used to determine the rationality index of the text generation system. Optionally, the rationality of the text generation system can be evaluated based on the rationality of all the second texts in the multiple groups of texts, or sampling (such as random sampling) or screening (such as screening out the results of processing errors or obvious outliers, etc.) can be performed on these second texts, and the rationality of the text generation system can be evaluated based on the rationality of the sampled or screened second texts.

[0066] The method 400 may have the same or similar embodiments as the aforementioned method 200 and have similar advantages, which will not be described in detail for the sake of brevity.

[0067] Figure 5 An example architecture diagram of an evaluation system 500 for a text generation system according to some embodiments of the present disclosure is schematically shown. Figure 5 As shown, the evaluation system 500 may include the system 300. The system 300 may sequentially obtain multiple groups of first texts and second texts from the database 510, and evaluate the rationality of each second text respectively, and then evaluate the rationality of the text generation system based on the rationality of these second texts.

[0068] Figure 6 Schematically shows an example block diagram of an apparatus 600 for evaluating text rationality according to some embodiments of the present disclosure. Figure 6 As shown, the apparatus 600 may include a first acquisition module 610 , a second acquisition module 620 , a first agent 630 , a second agent 640 , and an evaluation module 650 .

[0069] Specifically, the first acquisition module 610 can be configured to: acquire a first text and a second text, wherein the second text is generated by a text generation system based on the first text, and the second text includes at least one sub-text, each sub-text in the at least one sub-text has a corresponding text source identifier, and the text source identifier indicates the source text from which the corresponding sub-text originates; the second acquisition module 620 can be configured to: acquire at least one source text based on the data source identifier of at least one sub-text; the first agent 630 can be configured to: for each source text in the at least one source text, based on the content related to the text theme in the source text, generate a summary text corresponding to the source text, wherein the text theme is determined based on the first text; the second agent 640 can be configured to: for each sub-text in the at least one sub-text, based on the summary text of the source text corresponding to the sub-text, determine the rationality of the sub-text; the evaluation module 650 can be configured to: evaluate the rationality of the second text based on the rationality of each sub-text in the at least one sub-text.

[0070] In some embodiments, the apparatus 600 may further include a third agent 660, which may be configured to determine a text theme based on at least one keyword in the first text.

[0071] The apparatus 600 may be used to implement the method 200 described above, and the relevant details have been described in detail above, and for the sake of brevity, they will not be repeated here. The apparatus 600 may have the same features and advantages as described above with respect to the method 200.

[0072] Figure 7 An example block diagram of an evaluation device 700 for a text generation system according to some embodiments of the present disclosure is schematically shown. Figure 7 As shown, the apparatus 700 may include an acquisition module 710 , a text evaluation module 720 , and a system evaluation module 730 .

[0073] Specifically, the acquisition module 710 can be configured to: acquire multiple groups of texts, wherein each group of texts includes a first text and a second text, and the second text is generated by a text generation system based on the first text; the text evaluation module 720 can be configured to: for each group of texts in the multiple groups of texts, through the device 600, evaluate the rationality of the second text in each group of texts; the system evaluation module 730 can be configured to: evaluate the rationality of the text generation system based on the rationality of at least a part of the second text in the multiple groups of texts.

[0074] The apparatus 700 may be used to implement the method 400 described above, and the relevant details have been described in detail above, and for the sake of brevity, they will not be repeated here. The apparatus 700 may have the same features and advantages as described above with respect to the method 400.

[0075] It should be understood that the apparatus 600 or 700 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0076] Figure 8 An example block diagram of a computing device 800 according to some embodiments of the present disclosure is schematically shown, which may represent various types of computing devices that can be used to deploy the apparatus 600 or 700 or some modules in the apparatus 600 or 700 provided by the present disclosure.

[0077] As shown, the example computing device 800 includes a processing system 801, one or more computer-readable media 802, and one or more I / O interfaces 803 that are communicatively coupled to each other. Although not shown, the computing device 800 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include such as control and data lines.

[0078] Processing system 801 represents the functionality of performing one or more operations using hardware. Thus, processing system 801 is illustrated as including hardware elements 804 that may be configured as processors, functional blocks, and the like. This may include implementing application specific integrated circuits in hardware or other logic devices formed using one or more semiconductors. Hardware elements 804 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0079] The computer-readable medium 802 is illustrated as including a memory / storage device 805. The memory / storage device 805 represents a memory / storage device associated with one or more computer-readable media. The memory / storage device 805 may include a volatile storage medium (such as a random access memory (RAM)) and / or a non-volatile storage medium (such as a read-only memory (ROM), a flash memory, an optical disk, a disk, etc.). The memory / storage device 805 may include a fixed medium (e.g., RAM, ROM, a fixed hard drive, etc.) and a removable medium (e.g., a flash memory, a removable hard drive, an optical disk, etc.). Exemplarily, the memory / storage device 805 may be used to store the first text, the second text, etc. mentioned in the above embodiments. The computer-readable medium 802 may be configured in various other ways as further described below.

[0080] One or more input / output interfaces 803 represent functionality that allows a user to enter commands and information into the computing device 800 and also allows information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., a visible or invisible wavelength (such as infrared frequency) can be used to detect motion that does not involve touch as a gesture), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a display or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, an input device can be used to allow a user to provide a first text, a second text, or perform other interactive operations, and an output device can be used to allow a user to view text or evaluation results of a text generation system, and the like.

[0081] The computing device 800 also includes an evaluation application 806. The evaluation application 806 can be stored as computer program instructions in the memory / storage device 805. The evaluation application 806 can be implemented in conjunction with the processing system 801 and the like with respect to Figure 6 The functions of all or part of the modules of the device 600 for evaluating text rationality or the evaluation device 700 for a text generation system are described.

[0082] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Typically, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," etc., as used herein, generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0083] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 800. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.

[0084] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0085] "Computer readable signal media" refers to signal bearing media that is configured to send instructions to the hardware of computing device 800, such as via a network. Signal media typically can embody computer executable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave, data signal or other transmission mechanism. Signal media also includes any information transfer media. By way of example and not limitation, signal media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared and other wireless media.

[0086] As previously described, hardware elements 804 and computer-readable media 802 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0087] The aforementioned combination may also be used to implement various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 804. The computing device 800 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 804 of a processing system, a module may be implemented as a module executable by the computing device 800 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 800 and / or processing systems 801 to implement the techniques, modules and examples described herein.

[0088] The techniques described herein may be supported by these various configurations of computing device 800 and are not limited to the specific examples of the techniques described herein.

[0089] It should be understood that, for the sake of clarity, embodiments of the present disclosure are described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to a suitable unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0090] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the method for evaluating text rationality or the evaluation method for a text generation system is implemented.

[0091] The present disclosure provides a computer program product or a computer program, which includes computer executable instructions, which are stored in a computer-readable storage medium. A processor of a computing device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the computing device executes the method for evaluating text rationality or the evaluation method for a text generation system provided in the above various embodiments.

[0092] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for assessing the plausibility of text, include: Acquire a first text and a second text, wherein the second text is generated by a text generation system based on the first text, and the second text includes at least one subtext, each of the at least one subtext has a text source identifier, and the text source identifier indicates a source text from which the corresponding subtext originates; Based on the data source identifier of the at least one subtext, obtaining at least one source text; For each source text in the at least one source text, generating a summary text corresponding to the source text based on the content related to the text theme in the source text, wherein the text theme is determined based on the first text; For each sub-text of the at least one sub-text, determining the rationality of the sub-text based on a summary text of a source text corresponding to the sub-text; Based on the rationality of each sub-text in the at least one sub-text, the rationality of the second text is evaluated.

2. The method according to claim 1, further comprising: include: The text topic is determined based on at least one keyword in the first text.

3. The method according to claim 1, in, The step of generating a summary text corresponding to the source text based on the content related to the text theme in the source text comprises: Acquire at least one text block of the source text, wherein the at least one text block has a preset arrangement order; Initializing a current text block and a local summary text, wherein the current text block is initialized as the first text block in the at least one text block; Iterate through the following steps: Based on the content in the current text block that is related to the text topic, updating the local summary text; In response to the current text block not being the last text block in the at least one text block, updating the current text block to be the next text block according to the preset arrangement order; In response to the current text block being the last text block of the at least one text block, the summary text is determined based on the local summary text.

4. The method according to claim 3, in, The updating of the local summary text based on the content related to the text topic in the current text block includes: The local summary text and the content related to the text topic in the current text block are summarized, and the local summary text is updated based on the summary result.

5. The method according to claim 3, in, The obtaining of at least one text block of the source text comprises: The at least one text block is obtained based on at least one of the following items: a natural language segment in the source text, and a structured parsing result of the source text.

6. The method according to claim 1, in, The determining, for each sub-text of the at least one sub-text, the rationality of the sub-text based on the summary text of the source text corresponding to the sub-text comprises: For each subtext of the at least one subtext, a classification result of the subtext is determined according to a degree of association between the subtext and a summary text of a source text corresponding to the subtext, the classification result indicating whether the subtext is reasonable or unreasonable.

7. The method according to claim 6, in, The evaluating the rationality of the second text based on the rationality of each sub-text in the at least one sub-text includes: The rationality of the second text is evaluated based on the proportion of sub-texts classified as reasonable in the at least one sub-text.

8. An evaluation method for text generation systems, include: Acquire multiple groups of texts, wherein each group of texts includes a first text and a second text, and the second text is generated by a text generation system based on the first text; For each group of texts in the plurality of groups of texts, evaluating the rationality of a second text in each group of texts according to the method according to any one of claims 1 to 7; Based on the rationality of at least a portion of second texts in the plurality of groups of texts, the rationality of the text generation system is evaluated.

9. A device for evaluating the plausibility of a text, include: A first acquisition module is configured to: acquire a first text and a second text, wherein the second text is generated by a text generation system based on the first text, and the second text includes at least one subtext, each of the at least one subtext has a corresponding text source identifier, and the text source identifier indicates a source text from which the corresponding subtext originates; A second acquisition module is configured to: acquire at least one source text based on the data source identifier of the at least one subtext; A first agent is configured to: for each source text in the at least one source text, generate a summary text corresponding to the source text based on the content related to the text theme in the source text, wherein the text theme is determined based on the first text; A second intelligent agent is configured to: for each sub-text of the at least one sub-text, determine the rationality of the sub-text based on a summary text of a source text corresponding to the sub-text; The evaluation module is configured to evaluate the rationality of the second text based on the rationality of each sub-text in the at least one sub-text.

10. The device according to claim 9, further comprising: include: The third agent is configured to determine the text theme based on at least one keyword in the first text.

11. An evaluation device for a text generation system, include: An acquisition module is configured to: acquire multiple groups of texts, wherein each group of texts includes a first text and a second text, and the second text is generated by a text generation system based on the first text; A text evaluation module, configured to: for each group of texts in the plurality of groups of texts, evaluate the rationality of a second text in each group of texts by using the apparatus according to claim 10; The system evaluation module is configured to evaluate the rationality of the text generation system based on the rationality of at least a portion of the second text in the multiple groups of texts.

12. A computing device, include: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 8 when the computer executable instructions are executed by the processor.

13. A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to any one of claims 1 to 8.

14. A computer program product comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images