Event extraction system and method based on pre-trained model and word sense enhancement

Abstract

The application discloses an event extraction system and method based on a pre-training model and word sense enhancement, a domain word vector acquisition module obtains a domain word vector; a data set construction module obtains a specific domain text sequence data set; an event extraction model construction module constructs an event extraction model; a training module takes the domain word vector as an initial setting for training in a Soft-lexicon sub-model in the event extraction model, and trains the event extraction model by using the labeled specific domain text sequence data set; and an event extraction module predicts a label sequence result of the specific domain text sequence data set to be detected by using the trained event extraction model. By using the pre-training LERT sub-model and the Soft-lexicon sub-model, the application incorporates the domain vocabulary information of the domain word vector into the character representation, introduces a CRF layer in the model, improves the accuracy of label annotation in the event extraction task, and solves the problems of data scarcity, strong professionalism and complex context in specific domain event extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an event extraction system and method based on a pre-training model and word meaning enhancement. Background Art

[0002] Event extraction, as a key part of information extraction, aims to identify key elements describing events from text, such as trigger words, event subjects, event objects, and other related entities and attributes. It has important value in many application scenarios such as information retrieval, knowledge graph construction, and text classification.

[0003] In the task of extracting events in specific domains, we often face the problems of relatively scarce labeled data and relatively complex contexts. As for the relative scarcity of labeled data, the currently available event extraction datasets include the ACE2005 dataset and the Baidu DuEE dataset, but they are all general-domain datasets. There are relatively few datasets for texts in specific domains, making it difficult for the model to learn the unique features and patterns of data in specific domains through large amounts of data, thus affecting the accuracy of event extraction. For problems with more complex contexts, the terms in texts in specific domains are highly professional, and the semantic understanding of the terms requires domain knowledge as support. In addition, the context of texts is highly dependent, and event arguments are often closely related to contextual information. If the specific information of the vocabulary in the context is not taken into account during model processing, semantic errors or omissions will occur, affecting the accuracy of extraction.

[0004] Existing models face corresponding challenges when processing such text. On the one hand, existing deep learning models, especially neural network architectures, rely on large-scale, high-quality training data. However, the scarcity of data in specific fields limits the model's ability to learn complex semantic information through large amounts of data. In addition, despite the large vocabulary of deep learning models, it is still difficult to fully cover all professional vocabulary and terminology in a specific field. As a result, the model frequently encounters unknown words or terms when processing text, and it is difficult to accurately capture their precise meaning and semantic relationships in a specific context. On the other hand, although existing research has enhanced the model's ability to understand data to a certain extent by utilizing multi-dimensional feature representations (such as word vectors and term vectors), it is still insufficient when processing the complex semantic information of text in specific fields. These methods have difficulty effectively extracting the unique semantic features and contextual relationships in the text, thereby limiting the model's performance in tasks such as event extraction. Summary of the Invention

[0005] The purpose of the present invention is to provide an event extraction system and method based on a pre-trained model and word meaning enhancement. The present invention aims to mine valuable information from public text materials in a specific field. By constructing and training a corpus and model specifically for a certain field, the problems faced by traditional general models when processing events in a specific field, such as the relative scarcity of labeled data, the relatively complex context, and the lack of accuracy, are solved. By utilizing the Soft-lexicon sub-model initialization setting and combining it with a historical specific field text sequence data set for deep learning training, not only the accuracy of event extraction is improved, but also the understanding and adaptability to newly emerging text sequences are enhanced. The present invention can efficiently calculate and identify the label sequence combination with the highest probability in the text to be detected, thereby achieving accurate event extraction, improving information retrieval and analysis efficiency, providing strong support for research and decision-making in professional fields, and also promoting the development and optimization of natural language processing technology in specific application scenarios.

[0006] To achieve this goal, the present invention designs an event extraction system based on a pre-trained model and word meaning enhancement, comprising:

[0007] The domain word vector acquisition module is used to build a domain-specific corpus using publicly available textual materials from specific domain events. The domain word vector is obtained by training and evaluating the domain-specific corpus.

[0008] The dataset construction module is used to obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset;

[0009] The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework;

[0010] The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled domain-specific text sequence dataset to obtain a trained event extraction model.

[0011] The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

[0012] The beneficial effects of the present invention are:

[0013] 1) Use the pre-trained language model LERT to perform semantic understanding on domain-specific data to capture contextual information and semantic relationships in the text, so as to better adapt to the event extraction task.

[0014] 2) For professional terminology and domain knowledge in specific fields, word meaning enhancement technology is used to improve the model's ability to recognize these words, thereby improving the accuracy of the model's recognition and extraction of event elements.

[0015] 3) By fusing domain word vectors with character vectors generated by pre-trained models, richer semantic information is provided to help the model better capture the complex semantic relationships of texts in specific domains, thereby improving the accuracy of the model in identifying and extracting event elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a structural diagram of the event extraction system based on the pre-training model and word meaning enhancement of the present invention;

[0017] Figure 2 This is a structural diagram of the event extraction model based on the pre-training model and word meaning enhancement of the present invention;

[0018] Figure 3 This is a flow chart of language analysis of the LERT pre-training model of the present invention;

[0019] Figure 4 This is a pre-training flow chart of the LERT pre-training model of the present invention;

[0020] Figure 5 Schematic diagram of the Soft-lexicon sub-model in the present invention;

[0021] Figure 6 Schematic diagram of training loss value in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] Example 1:

[0024] like Figures 1 to 6 An event extraction system based on a pre-trained model and word meaning enhancement is shown, including:

[0025] The domain word vector acquisition module is used to build a domain-specific corpus using publicly available textual materials from specific domain events. The domain word vector is obtained by training and evaluating the domain-specific corpus.

[0026] The dataset construction module is used to obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset;

[0027] The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework;

[0028] The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled domain-specific text sequence dataset to obtain a trained event extraction model.

[0029] The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

[0030] The present invention proposes an event extraction system based on a pre-trained model and word meaning enhancement, constructing an efficient and accurate event extraction system. The main advantage of this system is that it can make full use of knowledge resources in specific fields, significantly improving the ability to understand complex contexts and the accuracy of event recognition.

[0031] The Domain Word Vector Acquisition module builds a highly targeted corpus from publicly available textual materials in a specific domain and trains word vectors from this corpus, which are rich in domain context information. These domain-specific word vectors serve as initial representations, providing a high-quality learning starting point for the model, accelerating training and enhancing generalization capabilities, enabling the model to better understand domain-specific terminology and technical language.

[0032] The dataset construction module converts the raw data format into a form suitable for sequence labeling tasks, ensuring data consistency and usability, laying a solid foundation for subsequent training. This step not only improves data quality but also facilitates the model's learning of structured information in the text, helping to improve the accuracy of event extraction.

[0033] In terms of model architecture, the feature extraction layer combines the LERT sub-model with the Soft-lexicon sub-model to achieve dual processing at both the character and lexical levels, resulting in a richer embedded representation. The feature fusion layer further integrates feature vectors at different levels, while the output decoding layer utilizes the Transformer model to capture long-range dependencies and integrate local features. Finally, the CRF layer calculates the probability of label sequence combinations and selects the optimal path as the output. This multi-level, multi-dimensional feature processing approach enhances the model's understanding of text content and improves the accuracy of event recognition.

[0034] In the training module, the parameters are initialized using pre-prepared domain word vectors, and iterative optimization is performed based on the annotated specific domain text sequence dataset, ensuring high stability and accuracy of the model in actual application scenarios. Through the evaluation of the validation set and the test set, the performance of the model is continuously adjusted to achieve the best state.

[0035] The event extraction module uses the trained model to predict the label sequence results based on the specific domain text sequence dataset to be detected, effectively converting the input text into structured event information. This method not only improves the quality of the event extraction task but also provides strong support for automated information extraction and analysis, suitable for various application scenarios that require precise event recognition, such as news report analysis, social media monitoring, and medical literature analysis. Overall, the system significantly improves the efficiency and accuracy of event extraction by deeply integrating domain knowledge and technical means.

[0036] In the above technical solution, the specific method for constructing the event extraction model is as follows:

[0037] The feature extraction layer processes the annotated specific domain text sequence dataset at the character level and the word level through the LERT sub-model and the Soft-lexicon sub-model in the event extraction model, obtaining character-level embedding representations and word-level embedding representations;

[0038] The feature fusion layer performs fusion and splicing operations on the character-level embedding representations and the word-level embedding representations to obtain a fused and spliced fusion vector;

[0039] The output decoding layer uses the Transformer model to capture the long-distance dependency relationships of characters in the fusion vector, integrates local features, and obtains label sequences for each character. The CRF layer is used to calculate the probability of each label sequence combination based on character combinations in the annotated specific domain text sequence dataset, forming an event extraction model.

[0040] Among them, the probability of the label sequence combination based on character combinations is selected as the result of the event extraction task of this patent.

[0041] In the above technical solution, the specific process for obtaining the domain word vector in the domain word vector acquisition module is as follows:

[0042] Using publicly available text materials of specific domain events, including Baidu Encyclopedia, news reports, and other data, a domain corpus is constructed. The domain corpus is cleaned and denoised to remove irrelevant information and stop words, and domain professional terms and common words are identified.

[0043] The cleaned corpus is trained using the Word2Vec model to obtain word vectors that can reflect the semantic characteristics of the domain;

[0044] The quality of word vectors is evaluated by similarity evaluation method.

[0045] Building a domain corpus provides the necessary data foundation for training domain-specific word vectors. These corpora not only serve as a data source for training word vector models, but also serve as a partial data source for building domain datasets. Furthermore, by collecting publicly available text materials in specific fields, the corpus is professional and targeted, which helps generate word vectors that better align with domain characteristics.

[0046] Cleaning and denoising are used to improve accuracy and term recognition, removing irrelevant information, stop words, and other noise, significantly improving the quality of word embeddings. Cleaned corpus focuses more closely on the core content within a domain, helping to generate word embeddings that better reflect the domain's semantic characteristics. It can also identify and retain both domain-specific terminology and common vocabulary, allowing the resulting word embeddings to better capture key concepts and expressions within the domain.

[0047] The Word2Vec model is used to capture semantic meaning, obtaining word vectors that reflect the semantic characteristics of a domain. These word vectors not only contain the co-occurrence relationship between words, but also implicitly contain the semantic information of the words in the specific context.

[0048] Similarity evaluation ensures that the generated word vectors accurately reflect the semantic features of a specific domain. High-quality word vectors can better capture lexical relationships and semantic information within the domain, thereby improving the performance of subsequent event extraction models. Similarity evaluation can screen out high-quality word vectors based on set evaluation metrics and thresholds. For example, using cosine similarity evaluation, word vectors with a cosine similarity greater than 0.8 are considered high-quality. However, in actual applications, it is necessary to select appropriate evaluation methods and metrics based on the specific application scenario and task requirements, and adjust and optimize them accordingly.

[0049] Domain word vectors are used in vocabulary-level embedding representations. During training, domain word vectors can be used as initial embedding layer parameters to help the model converge faster and improve model performance. Especially when processing text in a specific domain, domain word vectors can provide more accurate initial representations and accelerate the model's learning process.

[0050] In the above technical solution, the dataset construction module obtains specific event data and performs preprocessing. Preprocessing methods include removing irrelevant information, stop words, noise, and word segmentation. The preprocessed data is converted into a sequence annotation format, such as the BIO (Begin, Inside, Outside) format. "B-" indicates that the word is the beginning of an entity or argument; "I-" indicates that the word is part of an entity or argument but not the beginning; and "O-" indicates that the word does not belong to any entity or argument.

[0051] In the dataset construction module, the purity and accuracy of the data are ensured by obtaining data of specific events and performing preprocessing, including removing irrelevant information, stop words and noise, and segmenting words. The preprocessed data is converted into a sequence annotation format, which facilitates model understanding and learning, improves the accuracy and efficiency of the task, simplifies the subsequent data processing process, and enhances the effect and generalization ability of model training. In the above technical solution, in the event extraction model construction module, the feature extraction layer includes using the LERT sub-model and the Soft-lexicon sub-model to convert the specific field text sequence data into character-level embedding representations. char,i and word-level embedding representation e word,i , specifically:

[0052] The above LERT sub-model uses a multi-layer Transformer architecture as its core encoder, and uses the LERT sub-model to convert domain-specific text sequence data into character-level embedding representations. char,i The specific operation is: for the input sentence s consisting of k tokens (characters) = (c1, c2, ..., c k ), unique tags [CLS] and [SEP] are added to indicate the start and end of the sentence, and each token is concatenated into an embedding representation through character embedding, paragraph embedding, and position embedding, that is:

[0053]

[0054] Among them, E ci Represents the character c i The complete embedding representation, E token stands for character embedding, E seg represents paragraph embedding, E pos Represents position embedding.

[0055] Then sentence s=(c1,c2,…,c k ) is converted into a sequence of vectors Input into the bidirectional Transformer to obtain features. The Transformer's self-attention mechanism captures the context information of each token and generates a sequence vector e rich in semantic features char,i ={x1,x2,…,x k}.

[0056] Utilize the Soft-lexicon sub-model to convert domain-specific text sequence data into vocabulary-level embedding representations. word，i The specific process is: for the input text s=(c1,c2,…,c k ), when processing each character c i When a character matches a word in the dictionary, it is classified into the corresponding set. i The four sets are shown below:

[0057]

[0058]

[0059]

[0060]

[0061] Among them, L represents the dictionary, w m,n Indicates c m Start with c n End of the dictionary word, B(c i ) represents c i The starting word set, M(c i ) represents c i is the word set with the middle word, E(c i ) represents c i The word set ending with S(c i ) means only contains c i A set of single characters. If no words matching the {B,M,E,S} structure are found in the dictionary, the set is empty.

[0062] After constructing the character word set, these word sets are converted into vector representations with fixed dimensions by counting the frequency of each word in the word set, as shown in the following formula:

[0063]

[0064]

[0065] Among them, B, M, E, and S represent a word set respectively, and v s(S), v s (M), v s (E), v s (S) represents the weights of word sets B, M, E, and S respectively, z(w) represents the frequency of word w in the statistical word set, e w (w) represents the vector corresponding to word w in the domain word vector lookup table, Z represents the normalization factor, which is used to ensure that the weighted vocabulary information vector has an appropriate scale, e s (B,M,E,S) is the total vector representation of the four vocabulary sets B,M,E,S, e word,i ←{s,e s (B,M,E,S)}e word,i Represents the character vector after integrating vocabulary information, that is, vocabulary-level embedding.

[0066] The above process assigns each character a vector that integrates vocabulary information. In this way, each character in the input text sequence not only retains its original word vector but also incorporates the corresponding vocabulary information, achieving a one-to-one correspondence between the character and its vocabulary information.

[0067] The use of the soft-lexicon submodel described above is a simple and efficient method for incorporating lexical information into characters. This method requires only minor adjustments at the character representation layer to incorporate lexical information, eliminating the need for designing complex sequence modeling structures. This effectively simplifies model design while enhancing the model's ability to process lexical information. At the feature extraction layer, each character is mapped to a high-dimensional vector, and soft dictionary features are constructed, which are then integrated with the representation of each character. This approach effectively overcomes the shortcomings of models based solely on character information in utilizing lexical information. It also achieves better performance while maintaining a high inference speed, making it easier to integrate with pre-trained models such as BERT.

[0068] In the above technical solution, in the event extraction model construction module, the feature fusion layer fuses the character-level embedding representation and the word-level character representation, and the fused vector is represented as e fusion,i :

[0069]

[0070] in, Represents a vector concatenation operation.

[0071] The fusion layer combines the character-level embedding representation and the word-level embedding representation to fully utilize the semantic information at two different granularities.

[0072] The input text sequence obtains character-level embedding representation through the LERT sub-model, and adjusts and optimizes the word-level embedding representation through the Soft-lexicon sub-model. k ), the LERT sub-model converts each character c i Converted to an embedding vector e char,i , similarly, the Soft-lexiocn sub-model converts each word w i Converted to an embedding vector e word,i The fused representation is e fusion It can be expressed as:

[0073]

[0074] in, Represents a vector concatenation operation.

[0075] The fusion layer is a bridge connecting character representation and vocabulary representation, providing the model with more comprehensive and rich input features.

[0076] In the above technical solution, in the event extraction model construction module, the output decoding layer is structured as follows: encoder, decoder, and CRF layer, specifically:

[0077] The encoder is used to map the fusion vector provided by the feature fusion layer into a continuous semantic vector. The encoder includes a multi-head self-attention sublayer, a residual connection and normalization layer, and a feedforward fully connected network sublayer.

[0078] The decoder is used to gradually generate the label sequence of the output character according to the output of the encoder. The decoder includes a masked multi-head self-attention layer, an encoder-decoder self-attention layer, a feedforward fully connected network sublayer, a linear layer and a softmax function;

[0079] The CRF layer is used to learn the transition probability between labels based on the label sequence of the characters, obtain the probability of each label sequence combination, and take the label sequence combination with the highest probability as the final output.

[0080] The Transformer model is the core component of the output decoding layer, responsible for deeply understanding the bidirectional context of the input sequence and capturing the long-distance dependencies therein. Through its unique encoder-decoder structure, the Transformer model achieves efficient processing of event feature extraction tasks; the CRF layer can more accurately capture the dependencies between labels by learning the transition probabilities between labels, thereby improving the accuracy of annotation.

[0081] The output decoding layer is based on the Transformer model, and a CRF layer is introduced after the decoder of the Transformer model to further optimize the predicted label sequence.

[0082] The Transformer model includes an encoder and a decoder. The specific construction process is as follows:

[0083] Encoder: Each layer contains a multi-head self-attention sublayer, which captures the contextual information in the input sequence by calculating the attention weights between the query, key, and value. The formula is as follows:

[0084]

[0085] Among them, Q, K, and V represent query, key, and value respectively. k is the dimension of the key vector, Used to scale dot products to prevent the vanishing gradient problem. In this way, the model is able to focus on multiple positions in the input sequence in parallel, thereby gaining a more comprehensive understanding of the text content.

[0086] In order to enhance the stability and training efficiency of the model, the output of each sub-layer is added to its input (i.e., residual connection), and then layer normalization is performed. The formula is as follows:

[0087] LayerNorm(x+Sublayer(x));

[0088] Here, x is the input of the sublayer, Sublayer(x) is the output of the sublayer, and the addition operation represents the residual connection. This step ensures that the model can maintain a stable gradient flow during training and accelerates convergence.

[0089] After the multi-head self-attention sublayer, the encoder also includes a feed-forward fully connected network sublayer. This sublayer consists of two linear layers with a ReLU activation function in between for nonlinear transformation. The formula is as follows:

[0090] FFN(x)=max(0,xW1+b1)W2+b2;

[0091] Among them, W1 and W2 are weight matrices, and b1 and b2 are bias terms. This step further extracts high-dimensional features from the input sequence and provides richer semantic information for the subsequent decoder.

[0092] The Transformer's encoder is responsible for mapping the rich input feature representations provided by the feature fusion layer into continuous semantic vectors. These semantic vectors not only contain character-level and word-level embedding information, but also capture the dependencies between different positions in the input sequence through a multi-head self-attention mechanism.

[0093] Decoder: The decoder generates an output sequence (i.e., event elements) based on the encoder output. Each decoder layer consists of three sublayers: a masked multi-head self-attention layer, an encoder-decoder attention layer, and a feed-forward fully connected network.

[0094] Masked Multi-Head Self-Attention Layer: This sublayer ensures that the model does not depend on future outputs when generating a certain output, thus avoiding leakage of future information. This is crucial for sequence generation tasks because it ensures that the generated sequence is logical and grammatical.

[0095] Encoder-Decoder Attention Layer: This sublayer allows the decoder to focus on the encoder's output, thereby capturing the dependencies between the input and output sequences. By calculating the attention weights between the encoder output and the decoder's current state, the model is able to more accurately generate event features corresponding to the input sequence.

[0096] Feedforward fully connected network: The feedforward fully connected network sublayer in the decoder is similar to that in the encoder and is used to further extract and decode high-dimensional features in the input sequence.

[0097] Finally, the output of the last layer of the decoder is converted into a predicted probability distribution through a linear layer and a softmax function. These probability distributions correspond to the event elements in the text, and the model generates the final output sequence based on these probability distributions.

[0098] The Transformer decoder part gradually decodes the event elements corresponding to the input sequence.

[0099] However, relying solely on the output of the Transformer decoder cannot fully capture the dependencies between labels, especially in sequence labeling tasks. Therefore, after the last layer of the decoder, we introduce a CRF layer to further optimize the predicted label sequence.

[0100] The Transformer model combines the contextual information surrounding each position in a text sequence to enhance understanding of the current position. While maintaining sensitivity to long-range dependencies, the Transformer fully considers the elements or contextual information near each position in the sequence. This means that the output vector contains both information about the current position itself and information from other positions in the sequence, integrating local features and providing a richer and more accurate representation.

[0101] The construction process of the CRF layer is:

[0102] By learning the transition probabilities between labels, the CRF layer can more accurately capture the dependencies between labels, thereby improving the accuracy of annotation. During the decoding process, the CRF layer receives the output of the last decoder layer as input and calculates the probability of each possible label sequence combination. It then selects the label sequence combination with the highest probability as the final output.

[0103] Specifically, the CRF layer is optimized through the following steps:

[0104] Step 1. Calculate transition probabilities: The CRF layer first learns the transition probabilities between labels. These probabilities represent the likelihood of transitioning from one label to another.

[0105] Step 2. Calculate path probabilities: The CRF layer then calculates the probabilities of all possible label sequence combinations (i.e., paths). This is achieved by multiplying the output of the last decoder layer with the emission probability of the label (i.e., the probability that the decoder output corresponds to each label) and the transition probability between labels.

[0106] Step 3: Select the optimal path: The CRF layer selects the label sequence combination with the highest probability as the final output. This optimal path is the event feature sequence predicted by the model.

[0107] In the above technical solution, in the training module, the labeled specific domain text sequence data set includes a training set, a validation set and a test set. The training module also includes a verification and testing process. The specific construction method of the training module is: using the domain word vector as the initialization setting for model training, using the training set to train the training event extraction model, using the validation set and the test set to verify and test the trained model, outputting the predicted sequence, that is, the result of event extraction, and obtaining the precision, recall rate and F1 value.

[0108] Using domain word vectors as the initial setting for model training, or the initial representation, can help the model converge faster and improve model performance. Especially when processing text in a specific domain, domain word vectors can provide a more accurate initial representation and accelerate the model's learning process.

[0109] The training set is used to optimize model parameters to ensure it learns the correct patterns; the validation set is used to fine-tune hyperparameters and prevent overfitting, ensuring model generalization. Finally, the test set evaluates the model's true performance and outputs the predicted sequence, or event extraction results. By calculating metrics such as precision, recall, and F1 score, we can comprehensively measure model performance and ensure its reliability and effectiveness in real-world applications. This rigorous training and evaluation process not only improves model quality but also lays a solid foundation for subsequent application deployment.

[0110] The process of training the model on the training set is:

[0111] 1) Input the dataset D, word vector file gaz_file and pre-trained model file, set the model parameters and the maximum number of iterations Epoch;

[0112] 2) Initialize model parameters;

[0113] 3) Load the word vector file gaz_file;

[0114] 4) Load the pre-trained model file;

[0115] 5) Start training. For each iteration, read the training set D_train, divide it into multiple batches, and perform the following operations on each batch:

[0116] Convert text sequences into tensor format for model input;

[0117] Forward propagation calculates loss;

[0118] Backpropagate and update parameters;

[0119] Until the iteration is completed.

[0120] To better demonstrate the effectiveness and practicality of the event extraction system based on the pre-trained model and word meaning enhancement, as shown in Table 1, the MNEE dataset is used for experimental verification. The MNEE dataset is a military news event dataset constructed by the National University of Defense Technology based on public military news content. It contains a total of 8 event types and 10 types of event arguments.

[0121] Table 1 Data display of the model of the present invention compared with other models

[0122]

[0123] The three models of Lattice LSTM, LR-CNN and Flat-Lattice Transformer are all based on word information for semantic enhancement of sequence labeling method, and the core idea is to design a dynamic framework that can be compatible with word input, and the processing ability of the model for Chinese text is enhanced by introducing word-level information. The word information provides more rich context understanding for the model, which helps to more accurately identify event elements. From the experimental results, the effect after introducing the word information has improved to a certain extent compared with the basic LSTM-CRF.

[0124] Lattice LSTM fuses character-level and word-level information into the LSTM network by constructing a Lattice structure, improving the recognition ability of event elements. The experimental results show that Lattice LSTM exceeds LSTM-CRF in precision and recall, and the F1 value is improved by 7.95%, indicating that the introduction of word information can significantly enhance the recognition ability of the model for event trigger words.

[0125] LR-CNN improves the performance of event extraction by using CNN-based character feature encoding and word information fusion. The feature of the FLAT method is to add the boundary information of the Lattice word to the model by designing the position vector.

[0126] BERT-BiLSTM-CRF, ALBERT-BiLSTM-CRF and RoBERTa-BiLSTM-CRF are three methods for event extraction using pre-trained language models. The experimental results show that the introduction of pre-trained models improves the extraction effect to different degrees, and is better than the previous models in various indicators. This shows that the pre-trained model can provide rich semantic representation and enhance the understanding ability of the model for the text.

[0127] The model of the application combines the deep semantic understanding ability of the pre-trained language model and the semantic enhancement effect of the domain word information, and fuses information at the character level and the word level. This multi-level information processing helps the model to capture more detailed semantic differences, and effectively improves the semantic understanding ability of the model for the text in a specific field.

[0128] Embodiment 2

[0129] An event extraction method based on a pre-trained model and word meaning enhancement, comprising the following steps:

[0130] Step 1, using the public written materials of specific domain events, a specific domain corpus is constructed, and a domain word vector is obtained by training and evaluating the specific domain corpus;

[0131] Step 2, obtain a specific domain dataset, convert the specific domain dataset format to a sequence labeling format, and obtain an annotated specific domain text sequence dataset;

[0132] Step 3, taking the feature extraction layer, the feature fusion layer and the output decoding layer as the model framework, an event extraction model is constructed;

[0133] Step 4, inputting the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization setting, after the Soft-lexicon sub-model initialization setting, using the annotated specific domain text sequence dataset to train the event extraction model, obtaining the trained event extraction model;

[0134] Step 5, according to the specific domain text sequence dataset to be detected, using the trained event extraction model, calculating the probability of each character-based label sequence combination in the detection specific domain text sequence dataset, obtaining the highest probability label sequence combination in the detection specific domain text sequence dataset.

[0135] In step 1, by collecting and organizing public written materials of a specific domain, a high-quality domain corpus is constructed, and a specific domain word vector is trained. This step ensures that the model can understand the terminology and expression of the specific industry, and improves the understanding ability of professional text. The domain word vector captures the context relationship of the word in the domain, providing a more accurate representation basis for subsequent feature extraction, so that the model is more efficient in processing domain-related tasks.

[0136] In step 2, the collected specific domain text is converted into a sequence labeling format dataset, which can standardize the data structure and provide clear target labels for model learning. This format conversion allows each word or phrase to be associated with a specific event element, facilitating supervised learning by the model. The converted dataset can effectively guide the model to identify and classify key information in the text, improving the accuracy and efficiency of event extraction. In step 3, the feature extraction layer combines the LERT sub-model and the Soft-lexicon sub-model to process text sequences at the character level and the word level respectively, obtaining deep language feature representations. The LERT sub-model focuses on character-level details, helping to capture linguistic features, while the Soft-lexicon uses pre-trained domain word vectors to enhance semantic understanding at the word level. The combination of the two provides a rich and complementary source of information for subsequent fusion, greatly improving the model's ability to analyze complex language phenomena.

[0137] The feature fusion layer concatenates character-level and word-level embeddings to generate a fused vector, integrating the two different granularity feature representations and enhancing the model's representational capabilities. The fused vector incorporates both detailed character information and high-level lexical meaning, providing a more comprehensive and in-depth understanding for the output decoding layer. This approach effectively addresses the limitations of single-level features and enables the model to more accurately grasp text content.

[0138] The output decoding layer uses a Transformer model to capture long-range dependencies in the input fusion vector while integrating local features to ensure the consistency and logical coherence of the output sequence. A CRF (Conditional Random Field) layer is introduced to calculate the probability distribution of all possible label sequences, ultimately selecting the sequence with the highest probability as the prediction result. This operation not only improves the model's adaptability to complex syntactic structures but also enhances the precise location of event element boundaries, significantly improving the accuracy and reliability of event extraction.

[0139] In step 4, the domain word vectors are used as the initial representations for model training, and the event extraction model is trained using a known dataset, significantly improving the model's professionalism and accuracy. The domain word vectors pre-capture the semantic characteristics of specific industries, providing the model with rich initial weights and enabling it to understand specialized terminology and expressions from the outset. During training, these initial representations are combined with the ground-truth labels in the dataset to help the model more efficiently learn domain-specific patterns and rules. This not only accelerates convergence but also enhances the model's understanding and generalization capabilities for complex text, ensuring it can accurately identify and extract event elements in real-world applications, thereby providing more reliable and valuable analysis results. In step 5, the trained event extraction model is used to analyze the text sequence to be tested, predicting and outputting the event elements contained therein. This step puts all the preparatory work and model training results into practice, achieving the transition from theory to application. This approach can quickly locate and extract key event information from documents, providing valuable insights for users, and is widely applicable to various specific domains.

[0140] Example 3:

[0141] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0142] Example 4:

[0143] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the above method are implemented.

[0144] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art may still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

[0149] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. An event extraction system based on pre-training model and word meaning enhancement, characterized in that: include: The domain word vector acquisition module is used to build a domain-specific corpus using publicly available textual materials from specific domain events. The domain word vector is obtained by training and evaluating the domain-specific corpus. The dataset construction module is used to obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset; The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework; The feature extraction layer processes the labeled domain-specific text sequence dataset at the character level and the word level through the LERT sub-model and the Soft-lexicon sub-model in the event extraction model to obtain character-level embedding representation and word-level embedding representation; The feature fusion layer performs a fusion and splicing operation on the character-level embedding representation and the word-level embedding representation to obtain a fused and spliced ​​fusion vector; The output decoding layer captures the long-range dependencies of the characters in the fusion vector, integrates local features, obtains the label sequence of each character, and uses the CRF layer to calculate the probability of each label sequence combination based on the character combination in the annotated specific domain text sequence dataset to form an event extraction model; The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled domain-specific text sequence dataset to obtain a trained event extraction model. The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

2. The event extraction system based on pre-training model and word meaning enhancement according to claim 1 is characterized in that: In the domain word vector acquisition module, the specific process of obtaining the domain word vector is as follows: Utilize publicly available textual materials from specific domain events to construct a domain corpus, clean and denoise the domain corpus, remove irrelevant information and stop words, and identify domain-specific terminology and common vocabulary. The cleaned corpus is trained using the Word2Vec model to obtain word vectors that can reflect the semantic characteristics of the domain; The quality of word vectors is evaluated by similarity evaluation method.

3. The event extraction system based on pre-training model and word meaning enhancement according to claim 1 is characterized in that: In the event extraction model construction module, the feature extraction layer includes using the LERT sub-model and the Soft-lexicon sub-model to convert the domain-specific text sequence data into character-level embedding representations. and vocabulary-level embedding representations , where the LERT sub-model is used to convert to character-level embedding representation The specific method is: The acquired domain-specific text sequence data Mark the start and end, where express k characters; Each character is concatenated into an embedding representation through character embedding, paragraph embedding and position embedding, which is expressed as: ； in, Representative characters The complete embedding representation of represents character embedding, represents a paragraph embedding, represents positional embedding; Convert domain-specific text sequence data into vector sequences based on the concatenated embedding representation ; Use Transformer to obtain features and obtain character-level embedding representation ; The domain word vector is used as the initial representation of the trained Soft-lexicon sub-model to obtain the word-level embedding representation. , specifically: Domain-specific text sequence data Compare with the words in the dictionary and divide them accordingly to build a character word set. The specific division method is: ； ； ； ； Among them, L represents the dictionary, Indicates Start with The word in the dictionary that ends, Representatives The starting word set, represent A word set with the middle character. Representatives A collection of words ending with Represents only A set of single characters, if no matching character is found in the dictionary {B, M, E, S} The set is an empty set if the word has no structure. After constructing the character word set, the frequency of each word in the character word set is counted and converted into a vector representation with a fixed dimension to obtain the vocabulary level embedding , specifically expressed as: ； ； in, Represents a word set respectively. 、 、 、 Represents a set of words The weight of Representative words w The frequency of occurrence in the statistical word set, Representative words w The corresponding vector in the domain word vector lookup table, Represents the normalization factor, which is used to ensure that the weighted vocabulary information vector has an appropriate scale. Represented as four vocabulary sets B , M , E , S The total vector representation of , Represents the character vector after integrating vocabulary information, that is, vocabulary-level embedding.

4. The event extraction system based on pre-training model and word meaning enhancement according to claim 3 is characterized in that: In the event extraction model construction module, the feature fusion layer fuses the character-level embedding representation and the word-level character representation. The fused vector is represented as : ； in," " represents the concatenation operation of vectors.

5. The event extraction system based on pre-training model and word meaning enhancement according to claim 1 is characterized in that: In the event extraction model construction module, the output decoding layer is structured as follows: encoder, decoder, and CRF layer, specifically: The encoder is used to map the fusion vector provided by the feature fusion layer into a continuous semantic vector. The encoder includes a multi-head self-attention sublayer, a residual connection and normalization layer, and a feedforward fully connected network sublayer. The decoder is used to gradually generate the label sequence of the output character according to the output of the encoder. The decoder includes a masked multi-head self-attention layer, an encoder-decoder self-attention layer, a feedforward fully connected network sublayer, a linear layer and a softmax function; The CRF layer is used to learn the transition probability between labels based on the label sequence of the characters, obtain the probability of each label sequence combination, and take the label sequence combination with the highest probability as the final output.

6. According to the event extraction system based on pre-training model and word meaning enhancement in claim 1, in the training module, the labeled specific domain text sequence data set includes a training set, a validation set and a test set, and the training module also includes a verification and testing process. The specific construction method of the training module is: using the domain word vector as the initialization setting for model training, using the training set to train the training event extraction model, using the validation set and the test set to verify and test the trained model, outputting the predicted sequence, that is, the result of event extraction, and obtaining the precision, recall rate and F1 value.

7. An event extraction method based on pre-training model and word meaning enhancement, characterized in that: The steps include: The domain word vector acquisition module is used to build a domain-specific corpus using publicly available textual materials from specific domain events. The domain word vector is obtained by training and evaluating the domain-specific corpus. Obtain a domain-specific dataset, convert the domain-specific dataset format into a sequence annotation format, and obtain an annotated domain-specific text sequence dataset; An event extraction model is constructed using the feature extraction layer, feature fusion layer, and output decoding layer as the model framework; The feature extraction layer processes the labeled domain-specific text sequence dataset at the character level and the word level through the LERT sub-model and the Soft-lexicon sub-model in the event extraction model to obtain character-level embedding representation and word-level embedding representation; The feature fusion layer performs a fusion and splicing operation on the character-level embedding representation and the word-level embedding representation to obtain a fused and spliced ​​fusion vector; The output decoding layer captures the long-range dependencies of the characters in the fusion vector, integrates local features, obtains the label sequence of each character, and uses the CRF layer to calculate the probability of each label sequence combination based on the character combination in the annotated specific domain text sequence dataset to form an event extraction model; Input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled domain-specific text sequence dataset to obtain a trained event extraction model. According to the specific field text sequence dataset to be detected, the trained event extraction model is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

Images