A quality assessment method for multi-document summarization

Abstract

The present invention provides a method for evaluating the quality of multi-document summarization. The method comprises: constructing a question-answer pair, generating a question and its corresponding answer for each document as a reference answer; then, generating an answer to the question based on the summary currently being evaluated; and finally, obtaining a generation status value for the summary currently being evaluated based on whether the question can be answered, the accuracy of the answer, and whether the summary contains abnormal sentences. The present invention can analyze the generation quality of multi-document summaries without relying on a reference standard.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for evaluating the quality of multi-document summary generation. Background Art

[0002] In fields such as news reporting, knowledge dissemination, and information compilation, multiple documents from multiple sources often cover the same event, providing a wealth of perspectives and diverse understandings. However, the sheer volume of articles and text content can lead to fatigue for readers. This has fueled the demand for multi-document summarization. Currently, large language models are effective for document summarization. For multi-document summarization, a single-document summary can be generated and aggregated, and then directly output by a large language model with long context (such as Claude-3 or Gemini-1.5-pro), or by using Retrieval Augmented Generation (RAG). Current large language models with long context can handle sequences of hundreds of thousands of tokens. However, for multiple documents, the amount of information is vast, and the quality of content across documents varies, sometimes even containing conflicting information. Therefore, how to reasonably and objectively measure the quality of multi-document summaries generated by different models is a topic worth exploring. Most current summary evaluation methods focus on single-document summarization tasks with input content of several thousand tokens, using a reference summary-based approach. For multi-document summarization, the quality of the reference summary and its relevance based on human judgment are more important considerations. Generally speaking, in reference-based evaluation, the higher the overlap between a candidate summary and a standard reference summary, the better the summary quality. However, this evaluation method relies heavily on high-quality reference summaries, which are costly in multi-document tasks, and the correlation between automatic metrics and human judgment can also be problematic. Summary of the Invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] An embodiment of the present invention provides a method for evaluating the quality of multi-document summarization generation, the method comprising the following steps:

[0005] S100, obtaining a summary that currently needs to be evaluated; wherein the summary is a summary generated based on k documents related to the same event subject.

[0006] S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate the corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q1, A1), ..., (Q i , A i ),……,(Q n , A n )}, Q iis the i-th question in QAS, A i Q i The answer to the question is set, i ranges from 1 to n, and n is the number of question-answer pairs in the QAS; set question counter j=1, and execute S300.

[0007] S300, if j≤n, input the jth question and the summary currently to be evaluated into the large language model to obtain the answer to the jth question based on the summary currently to be evaluated. If the answer to the jth question is obtained, execute S400. If the answer to the jth question cannot be obtained, output the first reply identifier indicating that the answer to the jth question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set j=j+1, and execute S300; the initial value of the current first reply identifier set is empty; if j>n, execute S800.

[0008] S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and judge whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set. Otherwise, output the third reply identifier. The initial value of the current second reply identifier set is empty; set j=j+1 and execute S300.

[0009] S500 , splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting a sentence counter r=1, and executing S600 .

[0010] S600, if r≤m, input the rth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the rth sentence is a candidate abnormal sentence. If so, execute S700; if not, set r=r+1 and execute S600; if r>m, execute S800.

[0011] S700, input the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence. If so, add the rth sentence to the current target abnormal sentence set, set r=r+1, and execute S600; the initial value of the current target abnormal sentence set is empty.

[0012] S800 : Determine a generation status value corresponding to a summary currently requiring evaluation based on a current first reply identification set, a current second reply identification set, and a current target abnormal sentence set.

[0013] The present invention has at least the following beneficial effects:

[0014] The embodiment of the present invention provides a method for evaluating the quality of multi-document summary generation. This method obtains the generation status value of the summary by constructing question-answer pairs, without the need for costly reference summaries. It can achieve objective and accurate evaluation of the generation quality of multi-document summaries in a low-cost manner.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a flowchart of a method for evaluating the quality of multi-document summaries provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0020] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] The embodiment of the present invention provides a multi-document summary generation quality assessment method for analyzing the performance of a multi-document summary generation task of a large language model. Figure 1 As shown, the method may include the following steps:

[0022] S100, obtaining a summary that currently needs to be evaluated; wherein the summary is a summary generated based on k documents related to the same event subject.

[0023] In an embodiment of the present invention, the event subject may be determined based on actual circumstances. In one exemplary embodiment, the event subject may be, for example, news. k is an integer greater than 1, and its specific value may be determined based on actual circumstances. If the event subject is news, the k documents may be k news reports related to the news.

[0024] In an embodiment of the present invention, summaries of the k documents can be generated using an existing summary generation method, for example, using a large language model that has been trained to perform multi-document summary generation tasks. The large language model can be any existing large language model.

[0025] In an embodiment of the present invention, when a large language model is used to generate a summary of multiple documents, the large language model may be interacted with via a predetermined instruction statement (hereinafter referred to as interaction information) to generate summaries of k documents. In one exemplary embodiment, the following interaction information may be used to interact with the large language model to generate summaries of k documents:

[0026] ""

[0027] Below are multiple documents covering the same event. Please create a summary based on the content of these documents, making sure to include important information related to the core events, background, setting, characters, and their goals and motivations.

[0028]

Document 1

[0029] <document 1>

[0030]

Document 2

[0031] <document 2>

[0032]

Document k

[0033] <document k>

[0034] "". Among them, document represents the document, and the content in the symbol "<>" is the content to be entered, for example<document k> Indicates that document k is entered in <>.

[0035] S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate the corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q1, A1), ..., (Q i , A i ),……,(Q n , A n )}, Q i is the i-th question in QAS, A i Q i The answer to the question is set, i ranges from 1 to n, and n is the number of question-answer pairs in the QAS; set question counter j=1, and execute S300.

[0036] For question generation, the first thing to ensure is the validity of the question, that is, questions related to the theme of the entire event must be generated to effectively serve as an analysis standard for the quality of summary generation. In an embodiment of the present invention, the preset question-answer pair generation instruction information includes: the number of question-answer pairs to be generated, the generation of questions based on the key information of the document, and the answers to the questions are obtained based on the corresponding document. That is, the questions generated by the large language model can explain the main points and core meaning of the current document, have answers that can be determined based on the current document, and can highlight important facts or evidence.

[0037] In one embodiment of the present invention, the number of question-answer pairs generated for each document may be different. The number of question-answer pairs generated for each document may be positively correlated with the length of the document. In another embodiment of the present invention, the number of question-answer pairs generated for each document may be the same. In an illustrative embodiment, the number p of question-answer pairs generated for each document satisfies the following condition: 3≤p≤5. When the number of question-answer pairs generated for each document is p, n=k×p.

[0038] In an embodiment of the present invention, a large language model may be interacted with through a set instruction statement to generate question-answer pair information for k documents. In an exemplary embodiment, the following interaction information may be used to interact with the large language model to generate question-answer pair information for k documents:

[0039] ""

[0040] Based on the given document, generate {} question-answer pairs. Each question should cover a key point in the document, and its answer should be directly derived from the document's content and involve important facts about the current event. Ensure that the questions are diverse, covering different topics and details. Please return in a standard list format: [['Question 1:...', 'Answer 1:...'], ['Question 2:...', 'Answer 2:...'], ...]

[0041]

document

[0042] <document>

[0043] ""

[0044] S300, if j≤n, input the jth question and the summary currently to be evaluated into the large language model to obtain the answer to the jth question based on the summary currently to be evaluated. If the answer to the jth question is obtained, execute S400. If the answer to the jth question cannot be obtained, output the first reply identifier indicating that the answer to the jth question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set j=j+1, and execute S300; the initial value of the current first reply identifier set is empty; if j>n, execute S800.

[0045] In the embodiment of the present invention, the large language model may be any existing large language model, and may be the aforementioned large language model for generating multi-document summaries.

[0046] In an embodiment of the present invention, if the large language model cannot obtain an answer to the jth question based on the processed data, it will output a first reply indicator. The first reply indicator can be set based on actual needs, for example, the first reply indicator can be "Null" or "No relevant information".

[0047] In an embodiment of the present invention, the following interaction information can be used to interact with the large language model to generate an answer to each question based on the summary:

[0048] ""

[0049] Read the following summary and answer the questions. If the given content provides the answer to the question, give the answer directly. If the given content does not contain relevant information, output "No relevant information". Return the result in a standard list format: ['1. Output of the first question', 'Output of the second question', ...]

[0050]

summary

[0051] <content>

[0052]

question

[0053] <q1>

[0054] <q2>

[0055] …

[0056] ""

[0057] Content represents the summary, and Q1 and Q2 represent the questions output by the large language model.

[0058] S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and judge whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set. Otherwise, output the third reply identifier. The initial value of the current second reply identifier set is empty; set j=j+1 and execute S300.

[0059] In the embodiment of the present invention, the answer in the QAS is used as a reference solution, and the large language model compares the answer to the j-th question obtained based on the data to be processed with the corresponding reference answer A j Perform semantic analysis to determine whether the semantics of the two are identical. If they are, a second reply flag is output; if they are not, a third reply flag is output. The second and third reply flags can be set based on actual needs. For example, the second reply flag can be "yes" and the third reply flag can be "no", or the second reply flag can be "yes" and the third reply flag can be "no", or the second reply flag can be "1" and the third reply flag can be "0", etc.

[0060] Those skilled in the art know that the method of using a large language model to perform semantic analysis to determine whether the semantics of two answers are the same belongs to the prior art.

[0061] In an embodiment of the present invention, the following interactive information can be used to interact with the large language model to determine whether two answers have the same meaning:

[0062] ""

[0063] Given a question, determine whether the following two answers have the same meaning. If they do, output 'yes'. If they do not, output 'no'.

[0064]

question

[0065] <qj>

[0066]

Reference answer

[0067] <aj>

[0068] Generate Answers

[0069] <xj>

[0070] ""

[0071] Among them, Qj represents the jth question, Aj is the reference answer to Qj, X j is the answer to the j-th question obtained based on the summary.

[0072] S500 , splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting a sentence counter r=1, and executing S600 .

[0073] In the embodiment of the present invention, the abstract is split into sentences according to the periods in the abstract.

[0074] S600, if r≤m, input the rth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the rth sentence is a candidate abnormal sentence. If so, execute S700; if not, set r=r+1 and execute S600; if r>m, execute S800.

[0075] In an embodiment of the present invention, the preset candidate abnormal sentence description information may include: sentence incoherence, sentence repetition, grammatical issues, and causal omission. Sentence incoherence may include: interruptions in the narrative flow, including but not limited to: sudden jumps between perspectives, time periods, or settings; poor transitions between sentences and / or paragraphs; sentences and paragraphs that seem out of place; and illogical sentence order or summary structure. Sentence repetition refers to the redundant repetition of similar information. Grammatical issues refer to confusing wording or expression. Causal omission refers to the lack or unclear cause or motivation of something.

[0076] S700, input the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence. If so, add the rth sentence to the current target abnormal sentence set, set r=r+1, and execute S600; the initial value of the current target abnormal sentence set is empty.

[0077] When judging whether a sentence belongs to a candidate abnormal sentence based on the preset candidate abnormal sentence description information, the large language model will use several sentences before and after a sentence as a connection to determine whether the current sentence meets the preset candidate abnormal sentence description information, but it is possible that in the entire abstract, some content later can provide an answer to the previous error. Therefore, in an embodiment of the present invention, preset target abnormal sentence screening condition information is set to further judge the candidate abnormal sentences obtained based on the preset candidate abnormal sentence description information to determine whether the obtained candidate abnormal sentence is really an abnormal sentence, which can avoid misjudgment. In an embodiment of the present invention, the preset target abnormal sentence screening condition information may include the following conditions:

[0078] Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences;

[0079] Condition 2: There are no related sentences associated with the candidate anomaly sentence in the summary.

[0080] In this embodiment of the present invention, the associated sentences of a candidate abnormal sentence are sentences that cause the candidate abnormal sentence to not conform to the preset candidate abnormal sentence description information. For example, if a sentence in an abstract has the description "because...", it is determined to be a candidate abnormal sentence. If a sentence several sentences after this sentence has the description "so...", then the description "so..." is considered an associated sentence of the description "because...".

[0081] In an embodiment of the present invention, the following interaction information may be used to interact with the large language model to determine whether a sentence is a target abnormal sentence:

[0082] ""

[0083] Given a summary of an event and each sentence in the summary, determine whether there is an "error" in each sentence. The types of errors are as follows:

[0084] 1. Incoherence: interruptions in the narrative flow, including but not limited to: abrupt jumps between perspectives, time periods, or settings; poor transitions between sentences or paragraphs; sentences and paragraphs that seem out of place; and illogical sentence order or summary structure.

[0085] 2. Repetition: Redundantly repeating similar information.

[0086] 3. Grammatical issues; confusing wording or expression, etc.

[0087] 4. Causal omission: The cause or motivation of something is missing or unclear.

[0088] If the sentence sr has the above problem, determine whether it meets the following two conditions:

[0089] Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences;

[0090] Condition 2: There are no related sentences associated with the candidate anomaly sentence in the summary.

[0091] If the current sentence contains an error that meets these two conditions, output the error type and its cause. If not, output 'none'.

[0092]

Abstract paragraph

[0093] <summary>

[0094]

Current sentence

[0095] <sr>

[0096] ""

[0097] Among them, Summary means summary.

[0098] S800 : Determine a generation status value corresponding to a summary currently requiring evaluation based on a current first reply identification set, a current second reply identification set, and a current target abnormal sentence set.

[0099] Furthermore, the generated status value corresponding to the summary currently to be evaluated satisfies the following conditions:

[0100] S=w1×(1-(1 / n)×p1)+w2×((1 / (n-p1))×p2)+w3×(p3 / m).

[0101] Among them, S is the generation status value of the summary that currently needs to be evaluated, w1 is the first preset weight, w2 is the second preset weight, w3 is the third preset weight, p1 is the number of first reply identifiers in the current first reply identifier set, p2 is the number of second reply identifiers in the current second reply identifier set, and p3 is the number of target abnormal sentences in the current target abnormal sentence set.

[0102] In an embodiment of the present invention, the generation status value of the abstract currently requiring evaluation is used to characterize the quality of the generation of the abstract currently requiring evaluation. A larger generation status value indicates a better generation quality of the abstract currently requiring evaluation, and a smaller generation status value indicates a worse generation quality of the abstract currently requiring evaluation.

[0103] In an embodiment of the present invention, (1-(1 / n)×p1) represents the score of the generation quality of the summary that currently needs to be evaluated on the coverage analysis dimension, ((1 / (n-p1))×p2) represents the score of the generation quality of the summary that currently needs to be evaluated on the authenticity analysis dimension, and (p3 / m) represents the score of the generation quality of the summary that currently needs to be evaluated on the fluency analysis dimension. w1, w2, and w3 are all values ​​greater than 0 and less than 1. The specific values ​​of w1, w2, and w3 can be determined based on the analysis dimension of the user's preference. For example, if the user prefers the coverage dimension, the value of w1 is greater than w2 and w3. In an illustrative embodiment, w1=w2=w3=1 / 3.

[0104] Furthermore, in this embodiment of the present invention, p can be obtained by the following steps:

[0105] S1, set event subject counter q=1;

[0106] S2, if q≤q0, set the question-answer pair counter t=1 and execute S3, otherwise, execute S10; q0 is the number of preset event subjects, which can be set based on actual needs.

[0107] S3, input the received f(q) documents related to the same event subject q into the large language model to generate a summary ASq of the f(q) documents.

[0108] S4, if t≤t0, input each document in f(q) documents and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain the corresponding t question-answer pair information set QAS q ={(Q q1 , A q1 ),……,(Q qu , A qu ),……,(Q qt , A qt )}, Q qu For QAS q The u-th question in A qu Q qu The answer to u ranges from 1 to t; set the question counter x=1 and execute S5; if t>t0, set q=q+1 and execute S2; t0 is the number of preset question-answer pairs, which can be an experience value.

[0109] S5, if x≤t, sum the x-th question and the summary AS q Input the large language model to generate a summary based on AS q Get the answer to the x-th question. If the answer to the x-th question is obtained, execute S6. If the answer to the x-th question is not obtained, output the first reply identifier indicating that the answer to the x-th question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set x=x+1, and execute S5; the initial value of the current first reply identifier set is empty; if x>t, execute S9.

[0110] S6, will be based on the summary AS q The answer to the xth question obtained is consistent with the QAS q The answer to question x in A qx Compare and judge based on summary AS q The answer to question x obtained is the same as A qx Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set. Otherwise, output the third reply identifier. The initial value of the current second reply identifier set is empty; set x=x+1 and execute S5.

[0111] S7, summary AS q Split the sentences to obtain g(q) sentences; set the sentence counter z=1 and execute S8;

[0112] S8, if z≤g(q), input the zth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the zth sentence is a candidate abnormal sentence. If so, execute S9; if not, set z=z+1 and execute S8; if z>g(q), execute S10.

[0113] S9, inputting the zth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the zth sentence is a target abnormal sentence. If so, adding the zth sentence to the current target abnormal sentence set, setting z=z+1, and executing S8; the initial value of the current target abnormal sentence set is empty;

[0114] S10, based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set, determine the summary AS q The current generated state value is added to the current generated state value set C corresponding to the event subject q q In the process, set t=t+1 and execute S5; the initial value of the current generation state value is empty.

[0115] S11, obtain C1 to C q0 The number of question-answer pairs corresponding to the maximum generated state value in is taken as p.

[0116] In the embodiment of the present invention, the specific implementation of S3 to S10 may refer to the specific implementation of S100 to S800 described above.

[0117] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0118] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0119] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0120] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.< / sr> < / summary> < / xj> < / aj> < / qj> < / q1> < / content> < / document>

Claims

1. A method for evaluating the quality of multi-document summarization, characterized in that: The method comprises the following steps: S100, obtaining a summary currently requiring evaluation; wherein the summary is generated based on k documents related to the same event subject; S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate the corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q1, A1), ..., (Q i , A i ),……,(Q n , A n )}, Q i is the i-th question in QAS, A i Q i The answer to i is 1 to n, and n is the number of question-answer pairs in the QAS; set question counter j = 1, and execute S300; S300: If j ≤ n, the jth question and the summary currently to be evaluated are input into the large language model to obtain an answer to the jth question based on the summary currently to be evaluated. If the answer to the jth question is obtained, S400 is executed. If the answer to the jth question cannot be obtained, a first reply identifier indicating that the answer to the jth question cannot be obtained is output, and the output first reply identifier is added to the current first reply identifier set, j is set to j + 1, and S300 is executed. The initial value of the current first reply identifier set is empty. If j > n, S800 is executed. S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and judge whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If so, output a second reply identifier and add the output second reply identifier to the current second reply identifier set. Otherwise, output a third reply identifier. The initial value of the current second reply identifier set is empty. Set j = j + 1 and execute S300. S500, splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting a sentence counter r=1, and executing S600; S600: If r≤m, the rth sentence and the preset candidate abnormal sentence description information are input into the large language model to determine whether the rth sentence is a candidate abnormal sentence. If so, execute S700; if not, set r=r+1 and execute S600; if r>m, execute S800; S700: Input the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence. If so, add the rth sentence to the current target abnormal sentence set, set r=r+1, and execute S600; the initial value of the current target abnormal sentence set is empty; S800 : Determine a generation status value corresponding to a summary currently requiring evaluation based on a current first reply identification set, a current second reply identification set, and a current target abnormal sentence set.

2. The method according to claim 1, characterized in that The generation status value corresponding to the summary currently to be evaluated meets the following conditions: S=w1×(1-(1 / n)×p1)+w2×((1 / (n-p1))×p2)+w3×(p3 / m); Among them, S is the generation status value of the summary that currently needs to be evaluated, w1 is the first preset weight, w2 is the second preset weight, w3 is the third preset weight, p1 is the number of first reply identifiers in the current first reply identifier set, p2 is the number of second reply identifiers in the current second reply identifier set, and p3 is the number of target abnormal sentences in the current target abnormal sentence set.

3. The method according to claim 1, characterized in that The preset question-answer pair generation instruction information includes: the number of question-answer pairs to be generated, questions generated based on key information in the document, and answers to the questions obtained based on the corresponding documents.

4. The method according to claim 1, wherein The preset candidate abnormal sentence description information includes: sentence incoherence, sentence repetition, grammatical problems and causal omission.

5. The method according to claim 1, wherein The preset target abnormal sentence screening condition information includes the following conditions: Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences; Condition 2: There are no related sentences associated with the candidate anomaly sentence in the summary.

6. The method according to claim 3, characterized in that The same number of question-answer pairs are generated for each document.

7. The method according to claim 6, characterized in that The number of generated question-answer pairs p for each document is obtained by the following steps: S1, set event subject counter q=1; S2, if q≤q0, set the question-answer pair counter t=1 and execute S3, otherwise, execute S10; q0 is the number of preset event subjects; S3, input the received f(q) documents related to the same event subject q into the large language model to generate a summary ASq of the f(q) documents; S4, if t≤t0, input each document in f(q) documents and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain the corresponding t question-answer pair information set QAS q ={(Q q1 , A q1 ),……,(Q qu , A qu ),……,(Q qt , A qt )}, Q qu For QAS q The u-th question in A qu Q qu The answer to u ranges from 1 to t; set the question counter x = 1 and execute S5; if t> t0, set q = q + 1 and execute S2; t0 is the number of preset question-answer pairs; S5, if x≤t, sum the x-th question and the summary AS q Input the large language model to generate a summary based on AS q Get the answer to the x-th question. If the answer to the x-th question is obtained, execute S6. If the answer to the x-th question is not obtained, output the first reply identifier indicating that the answer to the x-th question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set x=x+1, and execute S5. The initial value of the current first reply identifier set is empty. If x>t, execute S9. S6, will be based on the summary AS q The answer to the xth question obtained is consistent with the QAS q The answer to question x in A qx Compare and judge based on summary AS q The answer to question x obtained is the same as A qx Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier. The initial value of the current second reply identifier set is empty. Set x=x+1 and execute S5; S7, summary AS q Split the sentences to obtain g(q) sentences; set the sentence counter z=1 and execute S8; S8, if z≤g(q), input the zth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the zth sentence is a candidate abnormal sentence. If so, execute S9; if not, set z=z+1 and execute S8; If z>g(q), execute S10; S9, inputting the zth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the zth sentence is a target abnormal sentence. If so, adding the zth sentence to the current target abnormal sentence set, setting z=z+1, and executing S8; The initial value of the current target abnormal sentence set is empty; S10, based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set, determine the summary AS q The current generated state value is added to the current generated state value set C corresponding to the event subject q q In the middle; set t = t + 1, execute S5; the initial value of the current generation state value is empty; S11, C1 to C q0 The number of question-answer pairs corresponding to the maximum generated state value in is taken as p.

Images