Document-level event element extraction method and system based on pre-training language model

Abstract

The invention is suitable for the technical field of document processing, and provides a document-level event element extraction method and system based on a pre-training language model.The method comprises the following steps that a document is loaded and preprocessed, and the document is segmented into a plurality of text segments through a sliding window method; a word segmentation device corresponding to the pre-training language model is used for marking the text segments, and the text is converted into a token sequence; inputting the token sequences into a pre-training language model, encoding each fragment, fusing the document-level global context, and determining the final representation of each token sequence; event element boundaries are recognized from the final representation of the token sequence, and element types are determined; and taking all the obtained elements as nodes, constructing an element graph, modeling the element graph through a graph neural network, and determining an event structure relationship. According to the method, the global semantics of the long document can be better modeled, the context information of the document can be deeply understood, and the event element extraction quality is improved.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a method and system for extracting document-level event elements based on a pre-trained language model. Background Technology

[0002] In the field of natural language processing, event element extraction is a crucial task. Its core objective is to accurately identify the key components of an event from text, encompassing aspects such as event type, participants, time, and location. Traditional event element extraction methods mostly focus on the sentence level, using individual sentences as the processing unit for element identification. However, in real-world applications, there are numerous long documents, such as news articles, financial reports, and medical records. These long documents possess complex structures and rich semantic information, and traditional sentence-level methods exhibit significant limitations when processing such documents.

[0003] On the one hand, event elements in long documents are often not concentrated in a single sentence, but scattered across multiple sentences or even different paragraphs. Traditional sentence-level extraction methods, focusing only on information within a single sentence and ignoring cross-sentence connections, are prone to missing or incomplete element extraction, i.e., information fragmentation. On the other hand, although pre-trained language models (such as BERT) have achieved significant results in natural language processing tasks, capturing local contextual information within a certain range, their application to document-level tasks is constrained by input length limitations. Long documents typically contain a large amount of textual information, far exceeding the upper limit of input length that pre-trained language models can handle. This makes it difficult for the model to model the global semantics of the document, thus affecting the accurate understanding and extraction of event elements.

[0004] Therefore, there is a need to provide a document-level event element extraction method and system based on pre-trained language models, aiming to solve the above problems. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a document-level event element extraction method and system based on a pre-trained language model, so as to solve the problems existing in the above-mentioned background technology.

[0006] This invention is implemented as follows: a document-level event element extraction method based on a pre-trained language model, the method comprising the following steps: Load and preprocess the document, then use a sliding window method to segment the document into multiple text fragments; The text fragments are tokenized using the tokenizer corresponding to the pre-trained language model, and the text is converted into a token sequence; The token sequence is input into a pre-trained language model, each segment is encoded, and the document-level global context is fused to determine the final representation of each token sequence; Identify event element boundaries and determine element types from the final representation of the token sequence; All obtained elements are used as nodes to construct an element graph. The element graph is then modeled using a graph neural network to determine the structural relationships of events. For each feature, a confidence score is calculated, and the event features and event structural relationships are converted into a structured format, including event ID, feature type, feature text, relationship type, and confidence score.

[0007] As a further aspect of the present invention: the step of encoding each segment and fusing the document-level global context to determine the final representation of each token sequence specifically includes: The context embedding of each token is calculated by a pre-trained language model, so that the representation of each token contains local context information; By processing the fragment sequence from left to right and from right to left using a bidirectional LSTM, an enhanced representation of each token is generated, which is then fused with the semantics of the entire document. The token's local context information and enhanced representation are fused together, and the final representation is generated through a fully connected layer.

[0008] As a further aspect of the present invention: the step of identifying event element boundaries and determining element types from the final representation of the token sequence specifically includes: Use sequence labeling models to identify feature boundaries at the token level; For the identified element span, classification is performed through aggregated representation: attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification, and negative sampling is used to adjust for class imbalance.

[0009] As a further aspect of the present invention: the step of modeling the element graph and determining the event structure relationships using a graph neural network specifically includes: The node representation is automatically updated iteratively through a graph neural network: each layer aggregates neighbor node information, enhances the representation through nonlinear transformation, and captures the semantic relationships between features; the updated feature representation is output, which includes the relational context. Based on the updated feature representation, each edge in the feature graph is classified into relationships: using the edge representation generation method, a fully connected classifier is input to predict the relationship type, which is predefined in the knowledge base, and the classifier outputs a probability distribution. A clustering algorithm is used to group events based on feature representation and relation type, outputting a structured list of events, each containing a set of features and relations.

[0010] As a further aspect of the present invention: when using a sequence labeling model to identify feature boundaries at the token level, a label for each token is predicted based on the final representation of each token. The label is used to indicate whether a feature is at its beginning, inside, or not.

[0011] As a further aspect of the present invention: when using a sequence labeling model, a global score for the entire label sequence is obtained, and the confidence score is calculated by comparing the score difference between the best sequence and the second-best sequence.

[0012] Another objective of this invention is to provide a document-level event element extraction system based on a pre-trained language model, the system comprising: The text fragment determination module is used to load and preprocess the document, and uses the sliding window method to divide the document into multiple text fragments; The text fragment processing module is used to tokenize text fragments using the tokenizer corresponding to the pre-trained language model, converting the text into a token sequence; The final representation generation module is used to input the token sequence into a pre-trained language model, encode each segment, and fuse the document-level global context to determine the final representation of each token sequence; The event element identification module is used to identify event element boundaries and determine element types from the final representation of the token sequence; The event structure relationship module is used to construct a feature graph by treating all obtained elements as nodes, and to model the feature graph using a graph neural network to determine the event structure relationship. The structured content output module is used to calculate the confidence score for each feature and convert the event features and event structural relationships into a structured format, including event ID, feature type, feature text, relationship type, and confidence score.

[0013] As a further aspect of the present invention: the final representation generation module includes: Local information unit, used to compute the context embedding of each token through a pre-trained language model, so that the representation of each token contains local context information; Full-text information units are used to process fragment sequences from left to right and from right to left using bidirectional LSTM, generating an enhanced representation of each token and fusing the semantics of the entire document; The fusion unit is used to fuse the local context information of the token with the enhanced representation, and generate the final representation through a fully connected layer.

[0014] As a further aspect of the present invention: the event element identification module includes: Feature boundary identification unit, used to identify feature boundaries at the token level using a sequence labeling model; The feature classification unit is used to classify the identified feature spans by aggregated representations: attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification, and negative sampling is used to adjust for class imbalance.

[0015] As a further aspect of the present invention: the event structure relationship module includes: The node representation update unit is used to automatically iteratively update the node representation through a graph neural network: each layer aggregates neighbor node information, enhances the representation through nonlinear transformation, and captures the semantic relationships between features; the output is the updated feature representation, which includes the relational context. The edge relationship classification unit is used to classify the relationship of each edge in the feature graph based on the updated feature representation: using the edge representation generation method, the fully connected classifier is input to predict the relationship type, the relationship type is predefined in the knowledge base, and the classifier outputs a probability distribution; The structured event list unit is used to group events based on feature representation and relation type using a clustering algorithm, outputting a structured event list where each event contains a set of features and relations.

[0016] Compared with the prior art, the beneficial effects of the present invention are: A sliding window approach is used to segment long documents into multiple text fragments, and overlapping tokens are used to alleviate information fragmentation to some extent. The token sequences are input into a pre-trained language model for encoding, and document-level global context is fused to determine the final representation of each token sequence. This allows for better modeling of the global semantics of long documents, enabling the model to understand the document's contextual information more deeply and thus improving the quality of event element extraction. By constructing a feature graph and using a graph neural network to model the feature graph, the structural relationships between event elements can be explicitly determined, resulting in a more complete extracted event structure that accurately reflects the logical and semantic relationships between various elements in the event, providing more comprehensive and accurate information for subsequent event analysis and applications. Attached Figure Description

[0017] Figure 1 This is a flowchart of a document-level event element extraction method based on a pre-trained language model.

[0018] Figure 2 This is a flowchart for determining the final representation in a document-level event element extraction method based on a pre-trained language model.

[0019] Figure 3This is a flowchart illustrating the process of determining the structural relationships of events in a document-level event element extraction method based on a pre-trained language model.

[0020] Figure 4 This is a schematic diagram of the structure of a document-level event element extraction system based on a pre-trained language model. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0023] like Figure 1 As shown, this embodiment of the invention provides a document-level event element extraction method based on a pre-trained language model, the method comprising the following steps: S100: Load the document and preprocess it, using the sliding window method to divide the document into multiple text fragments; S200 uses the tokenizer corresponding to the pre-trained language model to tokenize text fragments, converting the text into a token sequence; S300: Input the token sequence into the pre-trained language model, encode each segment, and fuse the document-level global context to determine the final representation of each token sequence; S400, Identify event element boundaries and determine element types from the final representation of the token sequence; S500 uses all obtained elements as nodes to construct an element graph, and models the element graph using a graph neural network to determine the structural relationships of events. S600 calculates a confidence score for each feature and converts event features and event structural relationships into a structured format, including event ID, feature type, feature text, relationship type, and confidence score.

[0024] It should be noted that traditional sentence-level event element extraction methods cannot effectively handle event elements scattered across multiple sentences in long documents, leading to omissions or incompleteness of elements and affecting the accuracy of event extraction. Pre-trained language models, limited by input length, struggle to model the global semantics of long documents when directly applied to document-level tasks, thus failing to fully understand the document's contextual information and impacting the accurate extraction of event elements. Existing methods extract elements in isolation, lacking explicit modeling of relationships between elements, resulting in incomplete event structures that cannot accurately reflect the true logic and semantic relationships of the events. The embodiments of this invention aim to solve the above problems.

[0025] In this embodiment of the invention, the document is first loaded and preprocessed. During preprocessing, regular expressions or natural language processing tools are used to remove irrelevant characters (such as HTML tags and special symbols) and standardize the text (such as unifying date formats). This step ensures clean text and reduces noise interference. Since pre-trained language models have a maximum input length limit (such as BERT's 512 tokens), a sliding window method is used to segment the document into multiple text fragments. Each text fragment is set to a length of N tokens, with M tokens overlapping, N=50, M=50. This ensures that information at the fragment boundaries is not lost, alleviating the information fragmentation problem to some extent. This method can better capture event elements scattered in different locations in long documents, improve the completeness and accuracy of element extraction, and avoid element omissions caused by information dispersion.

[0026] Then, the text fragments are tokenized using a pre-trained language model's corresponding tokenizer (such as BERT's WordPiece), converting the text into a token sequence. This token sequence is then input into the pre-trained language model (such as BERT) to encode each fragment and fuse it with the document-level global context to determine the final representation of each token sequence. This allows for better modeling of the global semantics of long documents, enabling the model to understand the document's contextual information more deeply and thus improving the quality of event element extraction. Next, event element boundaries are identified and element types are determined from the final representation of the token sequence. Here, a sequence labeling model (such as CRF) is used to predict a label based on the final representation of each token. Then, all the obtained elements are used as nodes to construct a feature graph. A graph neural network (GNN) is used to model the feature graph and determine the event structure relationships. This overcomes the shortcomings of existing methods that extract elements in isolation, resulting in a more complete extracted event structure that accurately reflects the logical and semantic relationships between various elements in the event, providing more comprehensive and accurate information for subsequent event analysis and applications. Finally, a confidence score is calculated for each element.

[0027] Specifically, when using a sequence labeling model, a global score for the entire label sequence is obtained. By comparing the score difference between the best and second-best sequences, a confidence score is calculated (the larger the score difference, the higher the confidence score). This converts event elements and event structural relationships into a structured format, including event ID, element type, element text, relationship type, and confidence score. This structured output facilitates subsequent storage, querying, and analysis, meeting the needs of different application scenarios for event information and improving the efficiency and accuracy of information processing.

[0028] like Figure 2As shown, in a preferred embodiment of the present invention, the steps of encoding each segment and fusing the document-level global context to determine the final representation of each token sequence specifically include: S301, calculate the context embedding of each token through a pre-trained language model, so that the representation of each token contains local context information; S302 processes the fragment sequence from left to right and from right to left using a bidirectional LSTM to generate an enhanced representation of each token and fuse the semantics of the entire document; S303 fuses the local context information of the token with the enhanced representation and generates the final representation through a fully connected layer.

[0029] Specifically, the fragment sequence is processed from left to right and from right to left using a bidirectional LSTM to generate an enhanced representation for each token, which is then fused with the semantics of the entire document. The specific steps are as follows: The sequence generated by segmenting text fragments using the sliding window method is input into the bidirectional LSTM network in the order in which they appear in the original document to obtain the initial hidden state sequence. Based on the initial hidden state sequence, using a gating mechanism, the text segments are processed one by one through the forward layer of a bidirectional LSTM in the natural reading order of the original document to obtain the forward hidden state of the token; Based on the initial hidden state sequence, the reverse gating computation process is used to process the text segments one by one in reverse order of the original document through the backward layer of the bidirectional LSTM to generate the backward hidden state of the token. The forward hidden state and the backward hidden state of the token are concatenated along the feature dimension to establish a cross-document contextual relationship based on the historical and future information of the current token in the original document, thereby obtaining the bidirectional hidden state of the token. Based on the bidirectional hidden state of the token, the enhanced hidden state of the token is obtained by using weight matrix calculation and nonlinear mapping mechanism, and through nonlinear processing of the fully connected layer to perform feature transformation. By leveraging the enhanced hidden state of the token, long-distance dependencies are captured through the pre-defined overlapping region in the sliding window method to obtain an enhanced representation of the global context; this enhanced representation of the global context is then used as an enhanced representation of the token.

[0030] Furthermore, this invention uses bidirectional LSTM to perform bidirectional sequence processing on the text fragments segmented by the sliding window, and combines forward and backward hidden states to effectively fuse the global semantics of the document; this invention utilizes overlapping regions to capture long-distance dependencies, enhances the contextual representation of each token, thereby improving the accuracy and completeness of event element extraction and overcoming the limitations of information dispersion in long documents.

[0031] In this embodiment of the invention, the pre-trained language model automatically calculates the context embedding of each token and obtains the hidden state output of each segment through forward propagation, wherein the representation of each token contains local context information. To capture the global context across segments, the computer introduces a document-level encoding layer (such as a bidirectional LSTM), and inputs the output embeddings of all segments into this layer in sequence, modeling the dependencies between segments through recurrent or self-attention mechanisms.

[0032] For example, a bidirectional LSTM processes the sequence of fragments from left to right and from right to left, generating an enhanced representation for each token and fusing it with the semantics of the entire document. Finally, the local context information of the tokens and the enhanced representations are fused using a weighted summation or concatenation method: for example, for each token, its local and global representations are concatenated and then passed through a fully connected layer to generate the final representation. This step ensures that the representation of each token includes both intra-fragment details and document-level context.

[0033] As a preferred embodiment of the present invention, the step of identifying event element boundaries and determining element types from the final representation of the token sequence specifically includes: Use sequence labeling models to identify feature boundaries at the token level; For the identified element span, classification is performed through aggregated representation. Attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification. Negative sampling is used to adjust and handle class imbalance.

[0034] Specifically, for the identified element span, classification is performed using aggregated representations: attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification. Negative sampling is used to adjust for class imbalance. The specific steps are as follows: Based on the event element boundaries, extract all token representations corresponding to the text span from the final representation to obtain a set of token representations; We utilize attention pooling to calculate the importance weight of each token in the token representation set in the feature classification task to achieve weighted fusion and obtain a pooled feature vector. The pooled feature vector is input into the fully connected layer. Through the linear transformation and non-linear activation processing of the fully connected layer, the dimension of the pooled feature vector is unified to obtain a fixed-dimensional aggregated vector. The fixed-dimensional aggregated vector is input into a fully connected classifier. The multi-layer neural network structure in the fully connected classifier performs hierarchical feature transformation on the fixed-dimensional aggregated vector to obtain a hierarchical feature transformation vector. Then, the hierarchical feature transformation vector is used to determine the feature type, and the softmax activation function of the output layer is used to generate the probability distribution of each feature type to obtain the probability distribution result. Based on the probability distribution results, the confidence score calculation method is used to extract the maximum probability value by comparing the difference between the confidence scores of the best and second-best sequences, so as to obtain the set of confidence scores; Based on the set of confidence scores and a preset probability threshold, a negative sampling adjustment method is used to handle class imbalance and to filter out all element types that exceed the preset probability threshold, thus obtaining a set of label types.

[0035] Furthermore, this invention uses attention pooling to weight and fuse all token representations within the feature span, highlighting key information and generating a feature vector with a unified dimension. It also combines a fully connected classifier with a softmax function to determine feature type, while introducing a negative sampling mechanism to effectively alleviate class imbalance. Finally, reliable results are selected based on confidence scores and probability thresholds, improving the accuracy and robustness of feature classification. In this embodiment of the invention, it is necessary to identify event element boundaries and classify their types from the enhanced token representations. Specifically, a sequence labeling model (such as CRF) is used to identify element boundaries at the token level. Based on the final representation of each token, a label for each token is predicted. The label is used to indicate whether the element is the start, inside, or not a element. The CRF layer considers the constraints between labels to improve boundary accuracy. For the identified element span (continuous token sequence), classification is performed through aggregation representation: attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification. The classifier uses a softmax or sigmoid activation function to output the element type probability (such as person, time, location), supporting multi-label classification (an element may belong to multiple types). To handle the problem of multiple element types, the computer sets a probability threshold (such as 0.5) to facilitate the automatic selection of all types exceeding the probability threshold as element labels. Negative sampling or loss functions are used to adjust for class imbalance, ensuring that rare element types can also be identified.

[0036] like Figure 3As shown, in a preferred embodiment of the present invention, the step of modeling the element graph and determining the event structure relationship through a graph neural network specifically includes: S501 automatically iteratively updates node representations through graph neural networks: each layer aggregates neighbor node information, enhances the representation through nonlinear transformations, and captures semantic relationships between features; the output is an updated feature representation that includes relational context. S502, based on the updated feature representation, classify the relationships of each edge in the feature graph; S503 uses a clustering algorithm to group elements based on their representation and relationship type, and outputs a structured list of events.

[0037] In this embodiment of the invention, relationships between elements are modeled, and an event structure is constructed to ensure the consistency and completeness of the extraction results. The initial edges of the element graph are generated based on element co-occurrence or syntactic dependencies: for example, edges are added between elements co-occurring in the same sentence or paragraph, or syntactic relationships are extracted using dependency parsing tools. The graph structure is stored in the form of an adjacency matrix for easy subsequent processing. A GNN is used to model the element graph, automatically iteratively updating the node representations: each layer aggregates neighbor node information, enhances the representation through nonlinear transformations, and captures semantic relationships between elements; the updated element representation is output, containing the relational context. Then, based on the updated element representations, each edge in the element graph is classified: using an edge representation generation method, a fully connected classifier is input to predict the relation type (e.g., "participant-event" or "time-location"), where the relation types are predefined in a knowledge base, and the classifier outputs a probability distribution. Finally, a clustering algorithm is used to group elements based on their representations and relation types, outputting a structured list of events, where each event contains a set of elements and relations.

[0038] Specifically, the node representation is automatically iteratively updated using a graph neural network: each layer aggregates neighbor node information, enhances the representation through nonlinear transformations, and captures the semantic relationships between elements. The specific steps are as follows: Step 701: Use the final representation of the token sequence as the initial representation of each node in the feature graph; determine the set of direct adjacent nodes of each node based on the adjacency relationship of the feature graph. Step 702: For each node in the feature graph, based on the initial representation of the node and the set of its direct neighbors, the first layer of the graph neural network is used to perform a weighted summation of the representations of each neighboring node through an attention mechanism to obtain the node representation after the first layer of aggregation. Step 703: Use the preset weight matrix in the fully connected layer to perform a nonlinear transformation on the node representation after aggregation in the first layer. At the same time, introduce nonlinear features through the activation function to adjust the feature dimension and enhance the semantics of the node representation after aggregation in the first layer, so as to obtain the node representation after the nonlinear transformation in the first layer. Step 704: Use the node representation after the first nonlinear transformation as the input of the graph neural network, and repeat the aggregation operation to obtain the node representation after the second aggregation. Step 705: Using a nonlinear activation function and a fully connected layer, the node representation after aggregation in the second layer is processed to enhance the discriminative ability of the node representation, and the node representation after nonlinear transformation in the second layer is obtained. Step 706: By iteratively executing steps 704 to 705, the representation of each node is gradually updated. After reaching a preset number of iterations, the final node representation is obtained; the final node representation is used as the updated element representation.

[0039] Furthermore, this invention utilizes multi-layer iterative aggregation of graph neural networks and dynamically fuses information from each node and its neighbors using an attention mechanism to effectively capture deep semantic relationships between elements. This invention also enhances the modeling ability of event structure relationships by combining nonlinear transformations to improve feature representation and gradually optimize node embedding, thereby achieving more accurate and complete extraction of event element relationships.

[0040] Specifically, based on the updated feature representation, each edge in the feature graph is classified into relationships: using an edge representation generation method, a fully connected classifier is input to predict the relationship type, where the relationship type is predefined in the knowledge base, and the classifier outputs a probability distribution. The specific steps are as follows: Based on the updated feature representation, the updated representation is extracted from the nodes at both ends of each edge in the feature graph, and the updated feature representations of the nodes at both ends are concatenated along the feature dimension to obtain the initial representation vector. The initial representation vector is input into the fully connected layer, and the initial representation vector is linearly transformed by the preset weight matrix in the fully connected layer to achieve the unification of feature dimensions while enhancing semantic representation capability, and to obtain the edge representation after linear transformation. By using a nonlinear activation function, the edge representation after linear transformation is processed, and nonlinear features are introduced to enhance the discriminative ability of the edge representation after linear transformation, thus obtaining the edge representation after nonlinear activation. The edge representation after nonlinear activation is transformed using a preset weight matrix to obtain a high-level edge feature representation. The high-level edge feature representation is input into a fully connected classifier, and the high-level edge representation is processed hierarchically through a multi-layer neural network to obtain a relation type discrimination vector. Apply the softmax activation function to the relation type discrimination vector to obtain the probability distribution of relation types; determine the predicted relation type of the edge based on the probability distribution of relation types.

[0041] Furthermore, this invention generates edge representations by splicing the features of the nodes at both ends of the edge, combines linear transformation and nonlinear activation to effectively integrate the semantic associations between nodes, then performs hierarchical feature extraction through a multi-layer network, uses softmax to generate the probability distribution of relationship types, and finally combines the confidence threshold to determine the relationship category, thereby improving the accuracy and reliability of relationship classification between event elements.

[0042] like Figure 4 As shown, this embodiment of the invention also provides a document-level event element extraction system based on a pre-trained language model, the system comprising: The text fragment determination module 100 is used to load and preprocess the document, and uses the sliding window method to divide the document into multiple text fragments. The text fragment processing module 200 is used to tokenize text fragments using the tokenizer corresponding to the pre-trained language model, and convert the text into a token sequence; The final representation generation module 300 is used to input the token sequence into the pre-trained language model, encode each segment, and fuse the document-level global context to determine the final representation of each token sequence; Event element identification module 400 is used to identify event element boundaries and determine element types from the final representation of the token sequence; The event structure relationship module 500 is used to construct a feature graph by using all the obtained elements as nodes, and to model the feature graph through a graph neural network to determine the event structure relationship. The Structured Content Output Module 600 is used to calculate the confidence score for each feature and convert the event features and event structural relationships into a structured format, including event ID, feature type, feature text, relationship type, and confidence score.

[0043] In a preferred embodiment of the present invention, the final representation generation module 300 includes: Local information unit, used to compute the context embedding of each token through a pre-trained language model, so that the representation of each token contains local context information; Full-text information units are used to process fragment sequences from left to right and from right to left using bidirectional LSTM, generating an enhanced representation of each token and fusing the semantics of the entire document; The fusion unit is used to fuse the local context information of the token with the enhanced representation, and generate the final representation through a fully connected layer.

[0044] In a preferred embodiment of the present invention, the event element identification module 400 includes: Feature boundary identification unit, used to identify feature boundaries at the token level using a sequence labeling model; The feature classification unit is used to classify the identified feature spans by aggregated representations: attention pooling is used to merge all token representations within the span into a fixed-dimensional vector, which is then input into a fully connected classifier for classification, and negative sampling is used to adjust for class imbalance.

[0045] In a preferred embodiment of the present invention, the event structure relationship module 500 includes: The node representation update unit is used to automatically iteratively update the node representation through a graph neural network: each layer aggregates neighbor node information, enhances the representation through nonlinear transformation, and captures the semantic relationships between features; the output is the updated feature representation, which includes the relational context. The edge relationship classification unit is used to classify the relationship of each edge in the feature graph based on the updated feature representation: using the edge representation generation method, the fully connected classifier is input to predict the relationship type, the relationship type is predefined in the knowledge base, and the classifier outputs a probability distribution; The structured event list unit is used to group events based on feature representation and relation type using a clustering algorithm, outputting a structured event list where each event contains a set of features and relations.

[0046] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0047] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0048] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0049] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A document-level event element extraction method based on a pre-trained language model, characterized by, The method comprises the following steps: loading a document and preprocessing, using a sliding window method to segment the document into multiple text segments; using a tokenizer corresponding to a pre-trained language model to tokenize the text segments, converting the text into a token sequence; inputting the token sequence into the pre-trained language model, encoding each segment, and fusing the document-level global context to determine the final representation of each token sequence; identifying event element boundaries and determining element types from the final representation of the token sequence; building an element graph by taking all obtained elements as nodes, modeling the element graph through a graph neural network to determine event structure relationships; for each element, calculating a confidence score, and converting event elements and event structure relationships into a structured format containing event ID, element type, element text, relationship type, and confidence score. 2.The document-level event element extraction method based on a pre-trained language model according to claim 1, wherein, The step of encoding each segment and fusing the document-level global context to determine the final representation of each token sequence specifically comprises: calculating the context embedding of each token through the pre-trained language model, so that the representation of each token contains local context information; processing the segment sequence from left to right and right to left through a bidirectional LSTM to generate an enhanced representation of each token, fusing the semantics of the entire document; fusing the local context information and the enhanced representation of the token to generate the final representation through a fully connected layer. 3.The document-level event element extraction method based on a pre-trained language model according to claim 2, characterized in that, The specific steps of processing the segment sequence from left to right and right to left through a bidirectional LSTM to generate an enhanced representation of each token, fusing the semantics of the entire document, are as follows: inputting the sequence segmented by the sliding window method into the bidirectional LSTM network in the order of appearance in the original document to obtain an initial hidden state sequence; using a gating mechanism, processing the text segments one by one through the forward layer of the bidirectional LSTM in the natural reading order of the original document to obtain the forward hidden state of the token according to the initial hidden state sequence; using a reverse gating calculation process, processing the text segments one by one through the backward layer of the bidirectional LSTM in the reverse order of the original document to generate the backward hidden state of the token according to the initial hidden state sequence; concatenating the forward hidden state of the token and the backward hidden state of the token in the feature dimension to establish a cross-document context link according to the historical information and future information of the current token in the original document, and then obtaining the bidirectional hidden state of the token; using a weight matrix calculation and a nonlinear mapping mechanism, performing nonlinear processing through a fully connected layer to obtain an enhanced hidden state of the token according to the bidirectional hidden state of the token; using the enhanced hidden state of the token, capturing long-distance dependency relationships through the preset overlap area in the sliding window method to obtain an enhanced representation of the global context; taking the enhanced representation of the global context as the enhanced representation of the token.

4. The pre-trained language model based document-level event element extraction method according to claim 3, characterized in that, The step of identifying event element boundaries and determining element types from the final representation of the token sequence specifically comprises: The sequence labeling model is used to identify the element boundary at the token level; For the identified element span, classification is performed through aggregated representation: attention pooling is used to combine all token representations within the span into a fixed-dimensional vector, which is input into a fully connected classifier for classification, and negative sampling is used to adjust the class imbalance. 5.The pre-trained language model based document-level event element extraction method according to claim 4, wherein, For the identified element span, classification is performed through aggregated representation: attention pooling is used to combine all token representations within the span into a fixed-dimensional vector, which is input into a fully connected classifier for classification, and negative sampling is used to adjust the class imbalance, the specific steps are as follows: According to the event element boundary, all token representations of the corresponding text span are extracted from the final representation to obtain a set of token representations; The importance weight of each token in the token representation set in the element classification task is calculated by attention pooling to achieve weighted fusion and obtain a pooling feature vector; The pooling feature vector is input into a fully connected layer, and the dimensions of the pooling feature vector are unified through linear transformation and nonlinear activation processing of the fully connected layer to obtain a fixed-dimensional aggregated vector; The fixed-dimensional aggregated vector is input into a fully connected classifier, and the fixed-dimensional aggregated vector is hierarchically transformed through the multi-layer neural network structure in the fully connected classifier to obtain a hierarchical feature transformation vector; then the hierarchical feature transformation vector is subjected to element type discrimination, and the probability distribution of each element type is generated through the softmax activation function of the output layer to obtain a probability distribution result; Based on the probability distribution result, the confidence score calculation method is used to extract the maximum probability value by comparing the difference in confidence scores between the optimal and suboptimal sequences to obtain a set of confidence scores; According to the confidence score set, in combination with a preset probability threshold, a negative sampling adjustment method is used to handle the class imbalance and filter out all element types that exceed the preset probability threshold to obtain a set of label types. 6.The document-level event element extraction method based on a pre-trained language model according to claim 5, wherein, Through the graph neural network, the steps of modeling the element graph and determining the event structure relationship are as follows: The node representation is automatically iteratively updated by the graph neural network: the neighbor node information is aggregated at each layer, the representation is enhanced through nonlinear transformation to capture the semantic relationship between elements; the updated element representation is output, which includes the relationship context; Based on the updated element representation, the relationship classification of each edge in the element graph is performed: the edge representation generation method is used to input the fully connected classifier to predict the relationship type, which is predefined in the knowledge base, and the classifier outputs a probability distribution; The clustering algorithm is used to group based on the element representation and the relationship type, and a structured event list is output, each event containing an element and a relationship set.

7. The pre-trained language model based document-level event element extraction method according to claim 6, characterized in that, The node representation is automatically iteratively updated by the graph neural network: the neighbor node information is aggregated at each layer, the representation is enhanced through nonlinear transformation to capture the semantic relationship between elements, and the specific steps are as follows: Step 701, the final representation of the token sequence is taken as the initial representation of each node in the element graph; the direct adjacent node set of each node is determined according to the adjacency relationship of the element graph; Step 702, for each node in the element graph, according to the initial representation of the node and the set of directly adjacent nodes, a first layer aggregation operation of the graph neural network is adopted, and the representation of each directly adjacent node is weighted and summed through an attention mechanism to obtain a first layer aggregated node representation; Step 703, using a preset weight matrix in the fully connected layer to perform nonlinear transformation on the first layer aggregated node representation, and introducing nonlinear features through an activation function to adjust the feature dimension and enhance the semantics of the first layer aggregated node representation, to obtain a first layer nonlinear transformed node representation; Step 704, taking the first layer nonlinear transformed node representation as the input of the graph neural network, repeating the aggregation operation to obtain a second layer aggregated node representation; Step 705, using a nonlinear activation function and a fully connected layer to process the second layer aggregated node representation to strengthen the discriminative ability of the node representation, to obtain a second layer nonlinear transformed node representation; Step 706, by iteratively executing steps 704 to 705, the representation of each node is updated step by step, and after reaching a preset number of iterations, a final node representation is obtained; The final node representation is taken as the updated element representation. 8.The pre-trained language model based document-level event element extraction method according to claim 7, characterized in that, Based on the updated element representation, the relationship classification of each edge in the element graph is performed: using an edge representation generation method, inputting a fully connected classifier to predict the relationship type, the relationship type being predefined in a knowledge base, the classifier outputting a probability distribution, the specific steps being as follows: According to the updated element representation, the updated element representation of the nodes at both ends of each edge in the element graph is extracted, and the updated element representations of the nodes at both ends are spliced in the feature dimension to obtain an initial representation vector; The initial representation vector is input into a fully connected layer, and a preset weight matrix in the fully connected layer is used to perform linear transformation on the initial representation vector to realize the unification of the feature dimension while enhancing the semantic representation ability, and a linearly transformed edge representation is obtained; Using a nonlinear activation function, the linearly transformed edge representation is processed to enhance the discriminative ability of the linearly transformed edge representation by introducing nonlinear features, to obtain a nonlinearly activated edge representation; Using a preset weight matrix, the nonlinearly activated edge representation is transformed to obtain a high-level edge feature representation; The high-level edge feature representation is input into a fully connected classifier, and the high-level edge representation is processed hierarchically through a multi-layer neural network to obtain a relationship type discriminative vector; The relationship type discriminative vector is applied to a softmax activation function to obtain a probability distribution of the relationship type; the predicted relationship type of the edge is determined according to the probability distribution of the relationship type. 9.The pre-trained language model based document-level event element extraction method according to claim 8, wherein, When using a sequence labeling model, a global score of the entire label sequence is obtained, and the confidence score is calculated by comparing the score difference between the optimal sequence and the suboptimal sequence.

10. A document-level event element extraction system based on a pre-trained language model, characterized by, The system applies the document-level event element extraction method based on the pre-trained language model according to any one of claims 1 to 9, and the system comprises: A text segment determination module for loading a document and preprocessing, and segmenting the document into multiple text segments using a sliding window method; The text segment processing module is configured to tokenize the text segments using a tokenizer corresponding to the pre-trained language model, and convert the text into a token sequence; The final representation generation module is configured to input the token sequence into the pre-trained language model, encode each segment, and fuse a document-level global context to determine a final representation of each token sequence; The event element identification module is configured to identify event element boundaries and determine element types from the final representation of the token sequence; The event structure relationship module is configured to construct an element graph by taking all obtained elements as nodes, model the element graph through a graph neural network, and determine event structure relationships; The structured content output module is configured to calculate a confidence score for each element, and convert the event elements and event structure relationships into a structured format, including an event ID, an element type, an element text, a relationship type, and a confidence score.

11. The pre-trained language model based document-level event element extraction system according to claim 10, wherein, The final representation generation module includes: A local information unit configured to calculate a context embedding of each token through the pre-trained language model, so that the representation of each token contains local context information; A full-text information unit configured to generate an enhanced representation of each token by processing the segment sequence from left to right and from right to left through a bidirectional LSTM, and fuse the semantics of the entire document; A representation fusion unit configured to fuse the local context information and the enhanced representation of the token, and generate a final representation through a fully connected layer; The event element identification module includes: An element boundary identification unit configured to identify element boundaries at the token level using a sequence labeling model; An element classification unit configured to classify the identified element span through an aggregated representation: merge all token representations within the span into a fixed-dimensional vector through attention pooling, input a fully connected classifier for classification, and use negative sampling adjustment to handle class imbalance; The event structure relationship module includes: A node representation update unit configured to automatically and iteratively update node representations through a graph neural network: aggregate neighbor node information at each layer, enhance the representation through a nonlinear transformation, capture semantic relationships between elements, and output updated element representations containing relationship contexts; An edge relationship classification unit configured to classify each edge in the element graph based on the updated element representations: use an edge representation generation method, input a fully connected classifier to predict a relationship type, and output a probability distribution of the relationship type predefined in a knowledge base; A structured event list unit configured to group based on element representations and relationship types using a clustering algorithm, and output a structured event list, each event containing an element and a relationship set.

Images