A method, device and equipment for evaluating document coherence, and a storage medium

Abstract

The application provides a document coherence and continuity evaluation method, device, equipment and storage medium. The event argument is filled one by one in a mask prediction manner by calling a pre-trained language model first, and the event instance is restored to an event description sentence. Then, the event relationship between two event description sentences is mapped to a set of association words, and the association word with the highest confidence is selected as the event relationship prediction result. The event relationship prediction result includes event argument continuity and event relationship coherence. Finally, according to the measurement and fusion of the event argument continuity and the event relationship coherence, the document coherence and continuity evaluation result is generated. Through the document coherence and continuity evaluation, the application scope of automatic composition evaluation is widened.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and storage medium for evaluating document coherence. Background Technology

[0002] Event relationships objectively exist between events and act on originally isolated sets of events. Event relationships can connect events discrete in text, forming an event relationship network and a topological context of event development. However, in current technology, it is not possible to further broaden the applicable question types and application scenarios of automatic essay assessment based on document coherence assessment.

[0003] In view of the above, this application is hereby submitted. Summary of the Invention

[0004] This invention discloses a method, apparatus, device, and storage medium for evaluating document coherence, aiming to broaden the applicable question types and application scenarios of automatic essay evaluation based on document coherence evaluation.

[0005] The first embodiment of the present invention provides a method for evaluating document coherence, including:

[0006] The pre-trained language model is invoked and the event arguments are filled one by one using mask prediction to restore the event instances into event description statements; The event relationship between two event description statements is mapped to a set of related words, and the related word with the highest confidence is selected as the event relationship prediction result. The event relationship prediction result includes the coherence of event arguments and the coherence of event relationship. Based on the measurement and fusion of the event argument coherence and the event relationship coherence, an evaluation result of document coherence is generated, wherein the event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events; and the event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs.

[0007] Preferably, the step of calling the pre-trained language model and filling in the event arguments one by one using masked prediction to restore the event instance into an event description statement specifically involves: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

[0008] Preferably, the step of mapping the event relationship between two event description statements to a set of related words, and selecting the related word with the highest confidence as the event relationship prediction result, specifically involves: The event description statements corresponding to two event instances are concatenated using masking tags by employing a Prompt learning approach. The masking markers are predicted using a pre-trained language model, and the connector with the highest confidence is selected and mapped to obtain the event relationship between the two events.

[0009] Preferably, the document coherence is the product of the event argument coherence and the event relation coherence.

[0010] A second embodiment of the present invention provides a device for evaluating document coherence, comprising: The event description statement restoration unit is used to call the pre-trained language model and fill in the event arguments one by one in the form of mask prediction to restore the event instance into an event description statement; The event relationship prediction result selection unit is used to map the event relationship between two event description statements into a set of related words, and select the related word with the highest confidence as the event relationship prediction result. The event relationship prediction result includes the event argument coherence and the event relationship connection. The document coherence evaluation result generation unit is used to generate document coherence evaluation results based on the measurement and fusion of event argument coherence and event relationship coherence. The event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events; the event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs.

[0011] Preferably, the event description statement restoration unit is specifically used for: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

[0012] Preferably, the event relationship prediction result selection unit is specifically used for: The event description statements corresponding to two event instances are concatenated using masking tags by employing a Prompt learning approach. The masking markers are predicted using a pre-trained language model, and the connector with the highest confidence is selected and mapped to obtain the event relationship between the two events.

[0013] Preferably, the document coherence is the product of the event argument coherence and the event relation coherence.

[0014] The third embodiment of the present invention provides a document coherence evaluation device, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a document coherence evaluation method as described in any of the above embodiments.

[0015] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a document coherence evaluation method as described in any of the above claims.

[0016] Based on the document coherence evaluation method, apparatus, device, and storage medium provided by this invention, the following steps are taken: First, a pre-trained language model is invoked to fill in event arguments one by one using mask prediction, restoring event instances to event description statements. Then, the event relationship between two event description statements is mapped to a set of related words. The related word with the highest confidence is selected as the event relationship prediction result. The event relationship prediction result includes event argument coherence and event relationship coherence. Finally, based on the measurement and fusion of the event argument coherence and event relationship coherence, an evaluation result of document coherence is generated. The event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events. The event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs. Through document coherence evaluation, the applicable question types and application scenarios of automatic essay evaluation are broadened. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a document coherence evaluation method provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of a module of a document coherence evaluation device provided in the second embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0024] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0025] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] This invention discloses a method, apparatus, device, and storage medium for evaluating document coherence, aiming to broaden the applicable question types and application scenarios of automatic essay evaluation based on document coherence evaluation.

[0027] The first embodiment of the present invention provides a method for evaluating document coherence, which can be executed by a document coherence evaluation device (hereinafter referred to as the evaluation device), specifically, by one or more processors within the evaluation device, to at least implement the following steps: S101, call the pre-trained language model and fill in the event arguments one by one in the way of mask prediction, and restore the event instance to the event description statement; In this embodiment, the evaluation device can be a desktop computer, laptop computer, server, workstation, or other terminal with data processing and analysis capabilities. The evaluation device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment are realized through the combination of the operating system and application software.

[0028] Specifically, in this embodiment: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

[0029] More specifically: Assumption For a by An event instance consisting of one argument and one trigger word has ,in, Representative events Trigger words, Indicates an event The first in An event-based argument, .right Except All elements are grouped by length, assuming The longest event argument length is ,have , ,in Indicates by medium length is A list of arguments.

[0030] Initialization event Corresponding event refactoring statement = , No. wheel( The process of filling in event arguments is as follows: Select The first list in the middle All arguments constitute a candidate argument set. , All arguments in the text have an argument length of 1. Then, a pre-trained language model was used to perform the totalization. The next prediction, let the first... The mask prediction template used for the second time is , Its generation rule is: in the composition The Adding the same number before each element is equivalent to... The masking mark , In particular, when At that time, The number of tails added is equal to The masking mark , .

[0031] To avoid the low accuracy problem caused by predicting multiple masking tokens simultaneously, the pre-trained language model predicts only one word of the argument at a time. exist middle The confidence formula for location is as follows:

[0032] in This indicates that the pre-trained language model is in The hidden layer vector output by the position. This represents the transpose of the input layer embedding weight matrix of a pre-trained language model. This is a function that converts input characters into their corresponding IDs in the dictionary of a pre-trained language model.

[0033] For length of Argument Use a greedy approach Secondary mask prediction and its in Confidence level of the mask location. Padding will be performed. Second-rate, Each time will select It has the highest confidence level Replace the masking marker at the corresponding position with the single word, and then... The remaining words in the middle The remaining occlusion markers are used for the next filling, and the pre-trained language model predicts arguments. Fill to The confidence level formula is as follows:

[0034] Finally, select all. The event argument with the highest confidence level obtained in the second prediction is filled in with its corresponding position in the event reconstruction statement to obtain a new... At the same time The list corresponding to this argument Remove it from the list. The event argument filling process will proceed. Wheel, until All lists in the middle are empty.

[0035] S102, map the event relationship between two event description statements into a set of related words, and select the related word with the highest confidence as the event relationship prediction result, wherein the event relationship prediction result includes event argument coherence and event relationship coherence; Specifically, in this embodiment: The event description statements corresponding to two event instances are concatenated using masking tags by employing a Prompt learning approach. The masking markers are predicted using a pre-trained language model, and the connector with the highest confidence is selected and mapped to obtain the event relationship between the two events.

[0036] More specifically: Assuming the event to be predicted and The corresponding event refactoring statement is and Generate input templates for event relationship prediction. In the form of: [ ][ ]

[0037] Define three sets of conjunctions. , and These correspond to three types of event relationships: accompanying, following, and triggering. The specific definitions of the three sets are as follows: V Accompany ={"accompanying", "at the same time", "suddenly", "as soon as", "immediately"} V Follow ={"Next", "Then", "After", "Then", "Later"} V Causal ={"cause", "lead to", "therefore", "induce", "generate"} Any of the three sets of conjunctions mentioned above Fill to The obtained event relationship prediction confidence levels are as follows:

[0038] To fill in conjunctions in a greedy manner The highest confidence level obtained for each character. Let... , and They are sets , and The three conjunctions in the text have confidence levels for predicting event relationships, and their confidence levels are as follows: , and .make

[0039] final and The event relationship determination function is as follows:

[0040] S103, Based on the measurement and fusion of the event argument coherence and the event relationship coherence, an evaluation result of document coherence is generated, wherein the event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events; the event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs.

[0041] In this embodiment, the document coherence is the product of the event argument coherence and the event relationship coherence.

[0042] Specifically, in this embodiment Given text and The corresponding set of event instances is and ,in , ; , . and The set of events between discourses is The constituent elements are in the form of [ , ],in , ,

[0043] .

[0044] For any , determination The condition for an argument to be a coherent event is that any event argument... There exists any ,have .

[0045] For any , determination The condition for an event to be coherent in terms of arguments is that, for any event arguments... There exists any ,have .

[0046] count and The total number of argument coherent events contained in it is .but and The function for determining the coherence of event arguments between events is as follows: ,

[0047] in and Represent and The number of event instances in the event.

[0048] and The function for determining the coherence of the event relationships between them is as follows: ,

[0049] represent The number of elements, i.e. and The number of event pairs that have an event relationship between documents.

[0050] and The function for determining the continuity between them is as follows: ,

[0051] The following example further illustrates the above implementation: Step 1.1: Given input E1 = {wandering, tiger, one day} for Trigger words, i.e. .right Except Group all external elements by length to get , where Sort1={}, Sort2=tiger, Sort3={one day}. The longest event argument length is .

[0052] Step 1.2, Initialization Corresponding event refactoring statement = =Strolling around.

[0053] First round of filling event arguments: due to It is empty, therefore select All arguments in the set constitute a candidate argument set. {Tiger}, used in the first prediction By browsing around, mask prediction can be obtained using the pre-trained model MacBert. , =0.581, therefore the first choice replace , .renew for Mask prediction can be obtained by using the pre-trained model MacBert. Therefore, the second choice replace , Thus we can obtain... Fill to Confidence The second prediction used The calculation process is similar to the first prediction, and the result is... Therefore, the arguments Fill to The masking marker position is used to obtain a new event refactoring statement. = and from Remove from .

[0054] Second round of filling event arguments: due to It is empty, therefore select All arguments in the set constitute a candidate argument set. The calculation process is similar to that of the first round of filling event arguments, and finally the event reconstruction statement is obtained. =

[0055] Step 2 is further specified as follows: Step 2.1: Predict the event and The corresponding event refactoring statement is and Generate input templates for event relationship prediction. In the form of: [ ][ ]

[0056] They are sets , and The three elements with the highest confidence levels are 0.231, 2.909, and 0.856, respectively. = .

[0057] In conclusion,

[0058] Step 3 is further specified as follows: Step 3.1, Given a text and The corresponding event set is and .in ,in , , , , , , .

[0059] in They are respectively The trigger word, i.e.

[0060] and The set of events between discourses is ,have ={[ , ], [ , ], [ , ]} Step 3.2: Statistical analysis through iteration and achievable =9, then we have =1

[0061] ,

[0062] Step 3.3: Statistical analysis through iteration and ,have =

[0063] ,

[0064] Step 3.4 and The function for determining coherence is as follows: , =1 therefore , It has continuity Therefore, it is not difficult to see that the document coherence evaluation method proposed in this embodiment can achieve good results in both event relationship prediction and document coherence evaluation model.

[0065] Please see Figure 2 The second embodiment of the present invention provides a device for evaluating document coherence, comprising: Event description statement restoration form 201 is used to call the pre-trained language model and fill in the event arguments one by one in the manner of mask prediction, so as to restore the event instance into an event description statement; The event relationship prediction result selection unit 202 is used to map the event relationship between two event description statements into a set of related words, and select the related word with the highest confidence as the event relationship prediction result. The event relationship prediction result includes event argument coherence and event relationship coherence. The document coherence evaluation result generation unit 203 is used to generate a document coherence evaluation result based on the measurement and fusion of event argument coherence and event relationship coherence. The event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events; the event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs.

[0066] Preferably, the event description statement restoration unit is specifically used for: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

[0067] Preferably, the event relationship prediction result selection unit is specifically used for: The event description statements corresponding to two event instances are concatenated using masking tags by employing a Prompt learning approach. The masking markers are predicted using a pre-trained language model, and the connector with the highest confidence is selected and mapped to obtain the event relationship between the two events.

[0068] Preferably, the document coherence is the product of the event argument coherence and the event relation coherence.

[0069] The third embodiment of the present invention provides a document coherence evaluation device, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a document coherence evaluation method as described in any of the above embodiments.

[0070] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a document coherence evaluation method as described in any of the above claims.

[0071] Based on the document coherence evaluation method, apparatus, device, and storage medium provided by this invention, the following steps are taken: First, a pre-trained language model is invoked to fill in event arguments one by one using mask prediction, restoring event instances to event description statements. Then, the event relationship between two event description statements is mapped to a set of related words. The related word with the highest confidence is selected as the event relationship prediction result. The event relationship prediction result includes event argument coherence and event relationship coherence. Finally, based on the measurement and fusion of the event argument coherence and event relationship coherence, an evaluation result of document coherence is generated. The event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events. The event relationship coherence is the proportion of the number of event pairs with event relationships in two document event sets to the total number of event pairs. Through document coherence evaluation, the applicable question types and application scenarios of automatic essay evaluation are broadened.

[0072] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the device for implementing a document coherence assessment. For example, the apparatus described in the second embodiment of the present invention.

[0073] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the document coherence evaluation method, connecting the various parts of the method using various interfaces and lines.

[0074] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a document coherence evaluation method. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0075] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0076] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0077] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for evaluating document coherence, characterized in that, include: The pre-trained language model is invoked and the event arguments are filled one by one using mask prediction to restore the event instances into event description statements; The event relationship between two event description statements is mapped to a set of related words. The related word with the highest confidence is selected as the event relationship prediction result, wherein the event relationship prediction result includes event argument coherence and event relationship coherence. Specifically, the event description statements corresponding to two event instances are concatenated using masking tags using a Prompt learning method. The masking tags are predicted by a pre-trained language model, and the related word with the highest confidence is selected and mapped to obtain the event relationship between the two events. Based on the measurement and fusion of the event argument coherence and the event relation coherence, an evaluation result of document coherence is generated, wherein the event argument coherence is the proportion of the number of events with coherent event arguments in two document event sets to the total number of events; the event relation coherence is the proportion of the number of event pairs with event relations in two document event sets to the total number of event pairs; and the document coherence is the product of the event argument coherence and the event relation coherence.

2. The method for evaluating document coherence according to claim 1, characterized in that, The process involves calling a pre-trained language model and filling in event arguments one by one using masked prediction to restore event instances into event description statements. Specifically: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

3. A device for evaluating document coherence, characterized in that, include: The event description statement restoration unit is used to call the pre-trained language model and fill in the event arguments one by one in the form of mask prediction to restore the event instance into an event description statement; The event relationship prediction result selection unit is used to map the event relationship between two event description statements into a set of related words, and select the related word with the highest confidence as the event relationship prediction result. The event relationship prediction result includes event argument coherence and event relationship coherence. Specifically, the event relationship prediction result selection unit is used to: concatenate the event description statements corresponding to two event instances using masking tags through Prompt learning; predict the masking tags using a pre-trained language model; select the linker with the highest confidence and map it to obtain the event relationship between the two events. The document coherence evaluation result generation unit is used to generate a document coherence evaluation result based on the measurement and fusion of event argument coherence and event relationship coherence. The event argument coherence is the proportion of events with coherent event arguments in two document event sets to the total number of events; the event relationship coherence is the proportion of event pairs with an event relationship in two document event sets to the total number of event pairs; and the document coherence is the product of the event argument coherence and the event relationship coherence.

4. The document coherence evaluation device according to claim 3, characterized in that, The event description statement restoration unit is specifically used for: S201, using the event trigger words in the event instance as the initial state of the event description reconstructed statement, grouping the event arguments in the event instance according to their own length, and selecting one event argument group as the candidate argument set in ascending order in each round; S202, the pre-trained language model is invoked to predict all positions to be filled in the current event description reconstruction statement for each argument in the candidate argument set, and the argument with the highest confidence in this round of prediction is selected to fill the corresponding position in the event description reconstruction statement. S203, Update the event reconstruction statement and remove the argument from the candidate argument set; S204, perform the next prediction, and repeat the above process until all arguments are filled.

5. A device for evaluating document coherence, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement a document coherence assessment method as described in any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement a document coherence evaluation method as described in any one of claims 1 to 2.

Images