Multi-document abstract generation quality evaluation method

Abstract

The invention provides a multi-document abstract generation quality evaluation method, which comprises the following steps of: constructing a question and answer pair mode, and generating a question and a corresponding answer as answer reference for each document; then, based on the abstract needing to be evaluated currently, generating an answer to the question; then, based on whether the question can be answered or not, the question answering accuracy and whether the abstract contains an abnormal sentence or not, a generation state value of the abstract needing to be evaluated currently is obtained; according to the method, the generation quality of the multi-document abstract can be analyzed without depending on a reference standard.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for evaluating the quality of multi-document summary generation. Background Art

[0002] In the fields of news reporting, knowledge popularization, information compilation, etc., there are usually multiple document records from multiple sources for the same event, providing rich viewpoints and different understandings. For readers, the huge number of articles and text content will cause fatigue. This stimulates the demand for multi-document summary generation. At present, large language models can effectively generate summaries for documents. For the generation of multi-document summaries, a single document summary can be generated and then summarized, and a long-context large language model (such as Claude-3 or Gemini-1.5-pro) can be directly output or retrieved enhanced generation (RAG). The current long-context large language model can handle sequences of hundreds of thousands of tokens. However, for the case of multiple documents, the amount of information is large, and the content quality in multiple documents is not uniform, and there may even be conflicting information. Therefore, how to reasonably and objectively measure the quality of multi-document summaries generated by different models is a topic worth exploring. Most of the current methods for summarization evaluation focus on single-document summaries and tasks with thousands of tags of input content, and judge based on reference summaries. For multi-document summaries, the quality of reference summaries and the relevance based on human judgment should be considered more. Generally speaking, in reference-based evaluation, the higher the overlap between a candidate summary and a standard reference summary, the better the summary quality. However, this evaluation method relies heavily on high-quality reference summaries, which are costly in multi-document tasks, and the correlation between automatic metrics and human judgment may also be problematic. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] An embodiment of the present invention provides a method for evaluating the quality of multi-document summaries, the method comprising the following steps:

[0005] S100, obtaining a summary that currently needs to be evaluated; wherein the summary is a summary generated based on k documents related to the same event subject.

[0006] S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q 1 , A 1 ), ..., (Q i , A i ), ..., (Q n , A n)}, Q i is the i-th question in QAS, A i Q i The answer to the question is, i ranges from 1 to n, and n is the number of question-answer pairs in the QAS; question counter j=1 is set, and S300 is executed.

[0007] S300, if j≤n, input the jth question and the summary currently to be evaluated into the large language model to obtain the answer to the jth question based on the summary currently to be evaluated. If the answer to the jth question is obtained, execute S400. If the answer to the jth question cannot be obtained, output a first reply identifier indicating that the answer to the jth question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set j=j+1, and execute S300; the initial value of the current first reply identifier set is empty; if j>n, execute S800.

[0008] S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and determine whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier, and the initial value of the current second reply identifier set is empty; set j=j+1 and execute S300.

[0009] S500 , splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting the sentence counter r=1, and executing S600 .

[0010] S600, if r≤m, input the rth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the rth sentence is a candidate abnormal sentence, if so, execute S700, if not, set r=r+1, execute S600; if r>m, execute S800.

[0011] S700, input the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence. If so, add the rth sentence to the current target abnormal sentence set, set r=r+1, and execute S600; the initial value of the current target abnormal sentence set is empty.

[0012] S800, based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set, determine the generation state value corresponding to the summary currently requiring evaluation.

[0013] The present invention has at least the following beneficial effects:

[0014] The multi-document summary generation quality assessment method provided by the embodiment of the present invention obtains the generation status value of the summary by constructing question-answer pairs, without the need for costly reference summaries, and can achieve objective and accurate assessment of the generation quality of multi-document summaries in a low-cost manner.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a method for evaluating the quality of multi-document summaries provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0020] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] The embodiment of the present invention provides a multi-document summary generation quality assessment method for analyzing the performance of a multi-document summary generation task of a large language model. Figure 1 As shown, the method may include the following steps:

[0022] S100, obtaining a summary that currently needs to be evaluated; wherein the summary is a summary generated based on k documents related to the same event subject.

[0023] In the embodiment of the present invention, the event subject may be determined based on actual conditions. In an exemplary embodiment, the event subject may be news, for example. k is an integer greater than 1, and the specific value may be determined based on actual conditions. When the event subject is news, the k documents may be k news reports related to the news.

[0024] In the embodiment of the present invention, the summaries of the k documents can be generated by an existing summary generation method, for example, by a trained large language model capable of performing multi-document summary generation tasks. The large language model can be any existing large language model.

[0025] In an embodiment of the present invention, when a multi-document summary is generated by a large language model, the large language model may be interacted with through a set instruction statement (hereinafter referred to as interaction information) to generate summaries of k documents. In an illustrative embodiment, the large language model may be interacted with through the following interaction information to generate summaries of k documents:

[0026] ""

[0027] Below are multiple documents covering the same event. Please generate a summary based on the content of these documents, making sure to include important information related to the core events, background, setting, characters, and their goals and motivations.

[0028]

Document 1

[0029] <document 1>

[0030]

Document 2

[0031] <document 2>

[0032]

Document k

[0033] <document k>

[0034] "". In which, document means document, and the content in the symbol "<>" is the content to be entered, for example<document k> Indicates that document k is entered in <>.

[0035] S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q 1 , A 1 ), ..., (Q i , A i ), ..., (Q n , A n )}, Q i is the i-th question in QAS, A i Q i The answer to the question is, i ranges from 1 to n, and n is the number of question-answer pairs in the QAS; question counter j=1 is set, and S300 is executed.

[0036] For question generation, the first thing to ensure is the validity of the question, that is, to generate questions related to the theme of the entire event, so that they can be effectively used as an analysis standard for the quality of summary generation. In an embodiment of the present invention, the preset question-answer pair generation instruction information includes: the number of question-answer pairs generated, questions generated based on the key information of the document, and the answers to the questions are obtained based on the corresponding documents, that is, the questions generated by the large language model can explain the main points and core meaning of the current document, have answers that can be determined based on the current document, and can highlight important facts or evidence.

[0037] In one embodiment of the present invention, the number of question-answer pairs generated for each document may be different. The number of question-answer pairs generated for each document may be positively correlated with the length of the document. In another embodiment of the present invention, the number of question-answer pairs generated for each document may be the same. In an illustrative embodiment, the number of question-answer pairs generated for each document, p, satisfies the following condition: 3≤p≤5. When the number of question-answer pairs generated for each document is p, n=k×p.

[0038] In an embodiment of the present invention, the large language model may be interacted with through a set instruction statement to generate question-answer pair information of k documents. In an exemplary embodiment, the large language model may be interacted with through the following interactive information to generate question-answer pair information of k documents:

[0039] ""

[0040] Please generate {} question-answer pairs based on the given document. Each question should cover a key point in the document, and its answer should come directly from the content of the current document and involve important facts of the current event. Make sure the questions are diverse, covering different topics and details. Please return in a standard list format: [['Question 1:...','Answer 1:...'],['Question 2:...','Answer 2:...'],...]

[0041]

document

[0042] <document>

[0043] ""

[0044] .

[0045] S300, if j≤n, input the jth question and the summary currently to be evaluated into the large language model to obtain the answer to the jth question based on the summary currently to be evaluated. If the answer to the jth question is obtained, execute S400. If the answer to the jth question cannot be obtained, output a first reply identifier indicating that the answer to the jth question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set j=j+1, and execute S300; the initial value of the current first reply identifier set is empty; if j>n, execute S800.

[0046] In the embodiment of the present invention, the large language model may be any existing large language model, and may be the aforementioned large language model for generating multi-document summaries.

[0047] In an embodiment of the present invention, if the large language model cannot obtain the answer to the jth question based on the data to be processed, a first reply identifier is output. The first reply identifier can be set based on actual needs, for example, the first reply identifier can be "Null" or "no relevant information".

[0048] In the embodiment of the present invention, the following interaction information may be used to interact with the large language model to generate an answer to each question based on the summary:

[0049] ""

[0050] Read the following summary and answer the questions. If the given content can find the answer to the question, give the answer directly. If there is no relevant information in the given content, output "no relevant information". Return in standard list format: ['1. Output of the first question', 'Output of the second question', ...]

[0051]

summary

[0052] <content>

[0053]

question

[0054] <q1>

[0055] <q2>

[0056] …

[0057] ""

[0058] .

[0059] Among them, Content represents the summary. Q1 and Q2 represent the questions output by the large language model.

[0060] S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and determine whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier, and the initial value of the current second reply identifier set is empty; set j=j+1 and execute S300.

[0061] In the embodiment of the present invention, the answer in the QAS is used as a reference solution, and the large language model compares the answer to the jth question obtained based on the data to be processed with the corresponding reference answer A j Perform semantic analysis to determine whether the semantics of the two are the same. If they are the same, output the second reply identifier; if they are not the same, output the third reply identifier. The second reply identifier and the third reply identifier can be set based on actual needs. For example, the second reply identifier can be "yes" and the third reply identifier can be "no", or the second reply identifier can be "yes" and the third reply identifier can be "no", or the second reply identifier can be "1" and the third reply identifier can be "0", etc.

[0062] Those skilled in the art know that the method of using a large language model to perform semantic analysis to determine whether the semantics of two answers are the same belongs to the prior art.

[0063] In the embodiment of the present invention, the following interactive information may be used to interact with the large language model to determine whether the meanings of two answers are the same:

[0064] ""

[0065] According to the given question, determine whether the following two answers have the same meaning. If they are the same, output 'yes'. If they are different, output 'no'.

[0066]

question

[0067] <qj>

[0068]

Reference answer

[0069] <aj>

[0070] Generate Answers

[0071] <xj>

[0072] ""

[0073] .

[0074] Among them, Qj represents the jth question, Aj is the reference answer to Qj, and X j is the answer to the jth question obtained based on the summary.

[0075] S500 , splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting the sentence counter r=1, and executing S600 .

[0076] In the embodiment of the present invention, the abstract is split into sentences according to the periods in the abstract.

[0077] S600, if r≤m, input the rth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the rth sentence is a candidate abnormal sentence, if so, execute S700, if not, set r=r+1, execute S600; if r>m, execute S800.

[0078] In an embodiment of the present invention, the preset candidate abnormal sentence description information may include: incoherent sentences, repeated sentences, grammatical problems and causal omissions. Among them, sentence incoherence may include: interruption of narrative flow, including but not limited to: sudden jumps between perspectives, time periods or settings; poor transitions between sentences and or paragraphs; sentences and paragraphs seem out of place; sentence order or summary structure is illogical. Sentence repetition refers to redundant repetition of similar information. Grammatical problems refer to confusing wording or expressions, etc. Causal omission refers to the lack or unclear cause or motivation of something.

[0079] S700, input the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence. If so, add the rth sentence to the current target abnormal sentence set, set r=r+1, and execute S600; the initial value of the current target abnormal sentence set is empty.

[0080] When judging whether a sentence belongs to a candidate abnormal sentence based on the preset candidate abnormal sentence description information, the large language model will use several sentences before and after a sentence as a connection to determine whether the current sentence meets the preset candidate abnormal sentence description information, but it is possible that in the entire abstract, some content later can answer the previous error. Therefore, in an embodiment of the present invention, preset target abnormal sentence screening condition information is set to further judge the candidate abnormal sentences obtained based on the preset candidate abnormal sentence description information to determine whether the obtained candidate abnormal sentence is really an abnormal sentence, which can avoid misjudgment. In an embodiment of the present invention, the preset target abnormal sentence screening condition information may include the following conditions:

[0081] Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences;

[0082] Condition 2: There are no related sentences in the summary that are associated with the candidate anomaly sentence.

[0083] In the embodiment of the present invention, the associated sentence of the candidate abnormal sentence is a sentence that makes the candidate abnormal sentence not conform to the preset candidate abnormal sentence description information. For example, if the description of a sentence in the abstract is "because...", it is judged as a candidate abnormal sentence, and the description of the sentence located several sentences after this sentence is "so...", then the description "so..." is the associated sentence of the description "because...".

[0084] In the embodiment of the present invention, the following interaction information may be used to interact with the large language model to determine whether a sentence is a target abnormal sentence:

[0085] ""

[0086] Given a summary of an event and each sentence in the summary, determine whether there is an "error" in each sentence. The types of errors are as follows:

[0087] 1. Incoherence: interruptions in the narrative flow, including but not limited to: sudden jumps between perspectives, time periods, or settings; poor transitions between sentences or paragraphs; sentences and paragraphs that seem out of place; and illogical sentence order or summary structure.

[0088] 2. Repetition: Repeating similar information redundantly.

[0089] 3. Grammatical issues; confusing wording or expression, etc.

[0090] 4. Causal omission: The cause or motivation of something is missing or unclear.

[0091] If the sentence sr has the above problem, determine whether it meets the following two conditions:

[0092] Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences;

[0093] Condition 2: There are no related sentences in the summary that are associated with the candidate anomaly sentence.

[0094] If the current sentence contains an error that meets these two conditions, output the error type and its cause. If not, output 'none'.

[0095]

Abstract paragraph

[0096] <summary>

[0097]

Current sentence

[0098] <sr>

[0099] ""

[0100] .

[0101] Among them, Summary means summary.

[0102] S800, based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set, determine the generation state value corresponding to the summary currently requiring evaluation.

[0103] Furthermore, the generation status value corresponding to the summary currently to be evaluated satisfies the following conditions:

[0104] S=w1×(1-(1 / n)×p1)+w2×((1 / (n-p1))×p2)+w3×(p3 / m).

[0105] Among them, S is the generation status value of the summary that currently needs to be evaluated, w1 is the first preset weight, w2 is the second preset weight, w3 is the third preset weight, p1 is the number of first reply identifiers in the current first reply identifier set, p2 is the number of second reply identifiers in the current second reply identifier set, and p3 is the number of target abnormal sentences in the current target abnormal sentence set.

[0106] In an embodiment of the present invention, the generation status value of the abstract currently to be evaluated is used to characterize the generation quality of the abstract currently to be evaluated. The larger the generation status value, the better the generation quality of the abstract currently to be evaluated. The smaller the generation status value, the worse the generation quality of the abstract currently to be evaluated.

[0107] In an embodiment of the present invention, (1-(1 / n)×p1) represents the score of the generation quality of the summary that currently needs to be evaluated in the coverage analysis dimension, ((1 / (n-p1))×p2) represents the score of the generation quality of the summary that currently needs to be evaluated in the authenticity analysis dimension, and (p3 / m) represents the score of the generation quality of the summary that currently needs to be evaluated in the fluency analysis dimension. w1, w2, and w3 are all values ​​greater than 0 and less than 1. The specific values ​​of w1, w2, and w3 can be determined based on the analysis dimension of the user's preference. For example, if the user prefers the coverage dimension, the value of w1 is greater than w2 and w3. In an illustrative embodiment, w1=w2=w3=1 / 3.

[0108] Furthermore, in the embodiment of the present invention, p can be obtained by the following steps:

[0109] S1, set event subject counter q=1;

[0110] S2, if q≤q0, set the question-answer pair counter t=1 and execute S3, otherwise, execute S10; q0 is the number of preset event subjects, which can be set based on actual needs.

[0111] S3, input the received f(q) documents related to the same event subject q into the large language model to generate a summary ASq of the f(q) documents.

[0112] S4, if t≤t0, input each document in the f(q) documents and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain corresponding t question-answer pair information sets QAS q ={(Q q1 , A q1 ), ..., (Q qu , A qu ), ..., (Q qt , A qt )}, Q qu For QAS q The u-th problem in A qu Q qu The answer, u takes a value from 1 to t; set the question counter x=1, and execute S5; if t>t0, set q=q+1, and execute S2; t0 is the number of preset question and answer pairs, which can be an empirical value.

[0113] S5, if x≤t, add the xth question and the summary AS q Input into the large language model to generate a summary based on AS q Get the answer to the x-th question. If the answer to the x-th question is obtained, execute S6. If the answer to the x-th question cannot be obtained, output the first reply identifier indicating that the answer to the x-th question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set x=x+1, and execute S5; the initial value of the current first reply identifier set is empty; if x>t, execute S9.

[0114] S6, will be based on the summary AS q The answer to the xth question obtained is consistent with the QAS q The answer to question x in A qx Compare and judge based on summary AS q The answer to the xth question obtained is the same as A qx Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier, and the initial value of the current second reply identifier set is empty; set x=x+1 and execute S5.

[0115] S7, Summary AS q Split the sentences to obtain g(q) sentences; set the sentence counter z=1, and execute S8;

[0116] S8, if z≤g(q), input the zth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the zth sentence is a candidate abnormal sentence. If so, execute S9; if not, set z=z+1 and execute S8; if z>g(q), execute S10.

[0117] S9, inputting the zth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the zth sentence is a target abnormal sentence, and if so, adding the zth sentence to the current target abnormal sentence set, setting z=z+1, and executing S8; the initial value of the current target abnormal sentence set is empty;

[0118] S10, determining the summary AS based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set q The current generated state value is added to the current generated state value set C corresponding to the event subject q q In the process, t=t+1 is set and S5 is executed. The initial value of the current generated state value is empty.

[0119] S11, get C 1 To C q0 The number of question-answer pairs corresponding to the maximum generated state value in is taken as p.

[0120] In the embodiment of the present invention, the specific implementation of S3 to S10 may refer to the specific implementation of S100 to S800 described above.

[0121] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0122] The embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / sr> < / summary> < / xj> < / aj> < / qj> < / q1> < / content> < / document>

Claims

1. A method for evaluating the quality of multi-document summarization, characterized in that: The method comprises the following steps: S100, obtaining a summary that currently needs to be evaluated; wherein the summary is a summary generated based on k documents related to the same event subject; S200, input each document and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain the question-answer pair information set QAS = {(Q1, A1), ..., (Q i , A i ), ..., (Q n , A n )}, Q i is the i-th question in QAS, A i Q i The answer of i ranges from 1 to n, and n is the number of question-answer pairs in the QAS; set the question counter j=1, and execute S300; S300, if j≤n, input the jth question and the summary currently to be evaluated into the large language model to obtain the answer to the jth question based on the summary currently to be evaluated, if the answer to the jth question is obtained, execute S400, if the answer to the jth question cannot be obtained, output a first reply identifier indicating that the answer to the jth question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set j=j+1, and execute S300; the initial value of the current first reply identifier set is empty; if j>n, execute S800; S400, compare the answer to the j-th question obtained based on the summary currently to be evaluated with the answer A of the j-th question in the QAS j Compare and determine whether the answer to the jth question obtained based on the summary currently to be evaluated is consistent with A j Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier, and the initial value of the current second reply identifier set is empty; set j=j+1, and execute S300; S500, splitting the abstract currently to be evaluated into sentences to obtain m sentences; setting the sentence counter r=1, and executing S600; S600, if r≤m, input the rth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the rth sentence is a candidate abnormal sentence, if yes, execute S700, if not, set r=r+1, execute S600; if r>m, execute S800; S700, inputting the rth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the rth sentence is a target abnormal sentence, and if so, adding the rth sentence to the current target abnormal sentence set, setting r=r+1, and executing S600; the initial value of the current target abnormal sentence set is empty; S800: Determine a generation status value corresponding to a summary that currently needs to be evaluated based on a current first reply identification set, a current second reply identification set, and a current target abnormal sentence set.

2. The method according to claim 1, characterized in that The generation status value corresponding to the summary currently to be evaluated meets the following conditions: S=w1×(1-(1 / n)×p1)+w2×((1 / (n-p1))×p2)+w3×(p3 / m); Among them, S is the generation status value of the summary that currently needs to be evaluated, w1 is the first preset weight, w2 is the second preset weight, w3 is the third preset weight, p1 is the number of first reply identifiers in the current first reply identifier set, p2 is the number of second reply identifiers in the current second reply identifier set, and p3 is the number of target abnormal sentences in the current target abnormal sentence set.

3. The method according to claim 1, characterized in that The preset question-answer pair generation instruction information includes: the number of question-answer pairs to be generated, questions to be generated based on key information in the document, and answers to the questions to be obtained based on the corresponding documents.

4. The method according to claim 1, characterized in that: The preset candidate abnormal sentence description information includes: sentence incoherence, sentence repetition, grammatical problems and causal omission.

5. The method according to claim 1, characterized in that: The preset target abnormal sentence screening condition information includes the following conditions: Condition 1: If the candidate abnormal sentences are not deleted or modified, the main line of events cannot be understood or the summary has incoherent sentences; Condition 2: There are no related sentences in the summary that are associated with the candidate anomaly sentence.

6. The method according to claim 3, characterized in that The same number of question-answer pairs are generated for each document.

7. The method according to claim 6, characterized in that The number of generated question-answer pairs p for each document is obtained by the following steps: S1, set event subject counter q=1; S2, if q≤q0, set the question-answer pair counter t=1 and execute S3, otherwise, execute S10; q0 is the number of preset event subjects; S3, input the received f(q) documents related to the same event subject q into the large language model to generate a summary ASq of the f(q) documents; S4, if t≤t0, input each document in the f(q) documents and the preset question-answer pair generation instruction information into the large language model to generate corresponding question-answer pair information, and obtain corresponding t question-answer pair information sets QAS q ={(Q q1 , A q1 ), ..., (Q qu , A qu ), ..., (Q qt , A qt )}, Q qu For QAS q The u-th problem in A qu Q qu The answer of u ranges from 1 to t; set the question counter x=1 and execute S5; if t>t0, set q=q+1 and execute S2; t0 is the number of preset question and answer pairs; S5, if x≤t, add the xth question and the summary AS q Input into the large language model to generate a summary based on AS q Get the answer to the x-th question. If the answer to the x-th question is obtained, execute S6. If the answer to the x-th question cannot be obtained, output the first reply identifier indicating that the answer to the x-th question cannot be obtained, and add the output first reply identifier to the current first reply identifier set, set x=x+1, and execute S5. The initial value of the current first reply identifier set is empty. If x>t, execute S9. S6, will be based on the summary AS q The answer to the xth question obtained is consistent with the QAS q The answer to question x in A qx Compare and judge based on summary AS q The answer to the xth question obtained is the same as A qx Are they the same? If they are the same, output the second reply identifier and add the output second reply identifier to the current second reply identifier set; otherwise, output the third reply identifier, and the initial value of the current second reply identifier set is empty; Set x=x+1 and execute S5; S7, Summary AS q Perform sentence splitting to obtain g(q) sentences; set sentence counter z=1, and execute S8; S8, if z≤g(q), input the zth sentence and the preset candidate abnormal sentence description information into the large language model to determine whether the zth sentence is a candidate abnormal sentence, if yes, execute S9, if not, set z=z+1, and execute S8; If z>g(q), execute S10; S9, inputting the zth sentence and the preset target abnormal sentence screening condition information into the large language model to determine whether the zth sentence is a target abnormal sentence, and if so, adding the zth sentence to the current target abnormal sentence set, setting z=z+1, and executing S8; The initial value of the current target abnormal sentence set is empty; S10, determining the summary AS based on the current first reply identification set, the current second reply identification set and the current target abnormal sentence set q The current generated state value is added to the current generated state value set C corresponding to the event subject q q In; set t = t + 1, execute S5; the initial value of the current generation state value is empty; S11, C1 to C q0 The number of question-answer pairs corresponding to the maximum generated state value in is taken as p.

Images