Event extraction system and method based on pre-training model and word meaning enhancement

Abstract

The invention discloses an event extraction system and method based on a pre-training model and word meaning enhancement. A field word vector acquisition module obtains a field word vector; the data set construction module obtains a specific field text sequence data set; the event extraction model construction module constructs an event extraction model; the training module takes the field word vector as initial setting of training in a Soft-lexcon sub-model in an event extraction model, and trains the event extraction model by utilizing a labeled specific field text sequence data set; and the event extraction module predicts a tag sequence result of the to-be-detected specific field text sequence data set by using the trained event extraction model. According to the method, the pre-trained LERT sub-model and the Soft-lexcon sub-model are utilized to bring the field word information of the field word vector into character representation, and the CRF layer is introduced into the model, so that the accuracy of label labeling in the event extraction task is improved, and the problems of scarcity of labeling data, high specialty and complex context in specific field event extraction are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an event extraction system and method based on a pre-training model and word meaning enhancement. Background Art

[0002] Event extraction, as a key part of information extraction, aims to identify key elements describing events from text, such as trigger words, event subjects, event objects and other related entities and attributes. It has important value in many application scenarios such as information retrieval, knowledge graph construction and text classification.

[0003] In the task of event extraction in a specific domain, we often face the problems of relatively scarce annotated data and complex context. As for the relative scarcity of annotated data, the currently public event extraction datasets include the ACE2005 dataset and the Baidu DuEE dataset, but they are all general domain datasets. There are few datasets for texts in specific domains, which makes it difficult for the model to learn the unique characteristics and rules of data in specific domains through a large amount of data, thus affecting the accuracy of event extraction. For problems with more complex contexts, the terms in texts in specific domains are highly professional, and the semantic understanding of the terms requires domain knowledge as support. In addition, the context dependence of the text is strong, and event arguments are often closely related to contextual information. If the specific information of the vocabulary in the context is not taken into account during model processing, semantic errors or omissions will occur, affecting the accuracy of extraction.

[0004] Existing models face corresponding challenges when processing such texts. On the one hand, existing deep learning models, especially neural network architectures, rely on large-scale, high-quality training data. However, the scarcity of data in specific fields limits the model's ability to learn complex semantic information through large amounts of data. In addition, despite the large size of the vocabulary of deep learning models, it is still difficult to fully cover all professional vocabulary and terminology in a specific field, resulting in the model frequently encountering unknown vocabulary or terminology when processing text, and it is difficult to accurately capture its precise meaning and semantic relationship in a specific context. On the other hand, although existing research has enhanced the model's ability to understand data to a certain extent by using multi-dimensional feature representations (such as word vectors, word vectors, etc.), it is still insufficient when processing complex semantic information in texts in specific fields. These methods are difficult to effectively extract the unique semantic features and contextual relationships in the text, thereby limiting the performance of the model in tasks such as event extraction. Summary of the invention

[0005] The purpose of the present invention is to provide an event extraction system and method based on a pre-trained model and word meaning enhancement. The present invention aims to mine valuable information from public text materials in a specific field. By constructing and training a corpus and model specifically for a certain field, the problems faced by traditional general models in processing events in a specific field, such as relatively scarce annotated data, complex context and insufficient accuracy, are solved. Utilizing the Soft-lexicon sub-model initialization setting and combining the historical specific field text sequence data set for deep learning training, not only the accuracy of event extraction is improved, but also the understanding and adaptability to newly emerging text sequences are enhanced. The present invention can efficiently calculate and identify the highest probability label sequence combination in the text to be detected, thereby achieving accurate event extraction, improving information retrieval and analysis efficiency, providing strong support for research and decision-making in professional fields, and also promoting the development and optimization of natural language processing technology in specific application scenarios.

[0006] To achieve this purpose, the present invention designs an event extraction system based on a pre-trained model and word meaning enhancement, comprising:

[0007] The domain word vector acquisition module is used to construct a domain-specific corpus using the text materials disclosed by events in a specific domain, and obtain domain word vectors by training and evaluating the domain-specific corpus;

[0008] The dataset construction module is used to obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset;

[0009] The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework;

[0010] The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled specific domain text sequence data set to obtain a trained event extraction model.

[0011] The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

[0012] The beneficial effects of the present invention are:

[0013] 1) Use the pre-trained language model LERT to perform semantic understanding on domain-specific data to capture contextual information and semantic relationships in the text, so as to better adapt to the event extraction task.

[0014] 2) For professional terms and domain knowledge in specific fields, word meaning enhancement technology is used to improve the model's recognition ability of these words, thereby improving the accuracy of the model's recognition and extraction of event elements.

[0015] 3) By integrating the domain word vectors with the character vectors generated by the pre-trained model, richer semantic information is provided to help the model better capture the complex semantic relationships of texts in specific fields, thereby improving the accuracy of the model in identifying and extracting event elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram of the event extraction system based on the pre-training model and word meaning enhancement of the present invention;

[0017] Figure 2 It is a structural diagram of the event extraction model based on the pre-training model and word meaning enhancement of the present invention;

[0018] Figure 3 It is a flow chart of language analysis of the LERT pre-training model of the present invention;

[0019] Figure 4 It is a pre-training flow chart of the LERT pre-training model of the present invention;

[0020] Figure 5 It is a schematic diagram of the Soft-lexicon sub-model in the present invention;

[0021] Figure 6 Schematic diagram of training loss value in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] Embodiment 1:

[0024] like Figures 1 to 6 An event extraction system based on a pre-trained model and word meaning enhancement is shown, comprising:

[0025] The domain word vector acquisition module is used to construct a domain-specific corpus using the text materials disclosed by events in a specific domain, and obtain domain word vectors by training and evaluating the domain-specific corpus;

[0026] The dataset construction module is used to obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset;

[0027] The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework;

[0028] The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled specific domain text sequence data set to obtain a trained event extraction model.

[0029] The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

[0030] The present invention proposes an event extraction system based on a pre-trained model and word meaning enhancement, and constructs an efficient and accurate event extraction system. The main advantage of this system is that it can make full use of knowledge resources in specific fields and significantly improve the ability to understand complex contexts and the accuracy of event recognition.

[0031] The domain word vector acquisition module uses public text materials in a specific field to build a highly targeted corpus and trains word vectors rich in domain background information. These domain-specific word vectors serve as initial representations, providing a high-quality learning starting point for the model, accelerating the training process and enhancing generalization capabilities, enabling the model to better understand domain-specific terminology and technical language.

[0032] The dataset construction module converts the original data format into a form suitable for sequence labeling tasks, ensuring the consistency and availability of the data and laying a solid foundation for subsequent training. This step not only improves the data quality, but also promotes the model to learn the structured information in the text, which helps to improve the accuracy of event extraction.

[0033] In terms of model architecture, the feature extraction layer combines the LERT sub-model and the Soft-lexicon sub-model to achieve dual processing at the character level and vocabulary level, thereby obtaining a richer embedding representation. The feature fusion layer further integrates feature vectors at different levels, while the output decoding layer uses the Transformer model to capture long-distance dependencies and integrate local features. Finally, the probability of label sequence combinations is calculated through the CRF layer, and the optimal path is selected as the output. This multi-level and multi-dimensional feature processing method enhances the model's understanding of text content and improves the accuracy of event recognition.

[0034] In the training module, the parameters are initialized using pre-prepared domain word vectors, and iterative optimization is performed based on a labeled domain-specific text sequence dataset to ensure the high stability and accuracy of the model in actual application scenarios. Through the evaluation of the validation set and the test set, the model performance is continuously adjusted to achieve the best state.

[0035] The event extraction module uses the trained model to predict the label sequence results based on the text sequence dataset of the specific field to be detected, and realizes the effective conversion from input text to structured event information. This method not only improves the quality of event extraction tasks, but also provides strong support for automated information extraction and analysis. It is suitable for various application scenarios that require accurate event identification, such as news report analysis, social media monitoring, and medical literature analysis. Overall, the system significantly improves the efficiency and accuracy of event extraction by deeply integrating domain knowledge and technical means.

[0036] In the above technical solution, the specific method of constructing the event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework is as follows:

[0037] The feature extraction layer processes the labeled domain-specific text sequence dataset at the character level and the word level through the LERT sub-model and the Soft-lexicon sub-model in the event extraction model to obtain the character-level embedding representation and the word-level embedding representation;

[0038] The feature fusion layer performs a fusion and splicing operation on the character-level embedding representation and the word-level embedding representation to obtain a fused vector.

[0039] The output decoding layer uses the Transformer model to capture the long-distance dependencies of characters in the fusion vector, integrate local features, and obtain the label sequence of each character. The CRF layer is used to calculate the probability of each label sequence combination based on character combinations in the labeled specific domain text sequence dataset to form an event extraction model.

[0040] Among them, the probability of the label sequence combination based on the character combination is obtained, and the label sequence combination with the highest probability is selected as the result of this patent event extraction task.

[0041] In the above technical solution, in the domain word vector acquisition module, the specific process of obtaining the domain word vector is as follows:

[0042] Use the public text materials of events in a specific field, including Baidu Encyclopedia, news reports and other data, to build a field corpus, clean and denoise the field corpus, remove irrelevant information and stop words, and identify field professional terms and common words;

[0043] The Word2Vec model is used to train the cleaned corpus to obtain word vectors that can reflect the semantic characteristics of the domain;

[0044] The quality of word vectors is evaluated by similarity evaluation method.

[0045] Constructing domain corpora provides the necessary data foundation for training domain-specific word vectors. These corpora are not only the data source for training word vector models, but also can be used as part of the data source for constructing domain datasets. In addition, by collecting public text materials in specific fields, the professionalism and pertinence of the corpus are ensured, which helps to generate word vectors that are more in line with domain characteristics.

[0046] Cleaning and denoising are used to improve accuracy and term recognition, remove irrelevant information, stop words and other noise, and significantly improve the quality of word vectors. The cleaned corpus is more focused on the core content in the field, which helps to generate word vectors that better reflect the semantic characteristics of the field; it can identify and retain domain-specific terms and common words, so that the final generated word vector can better capture important concepts and expressions in the field.

[0047] The Word2Vec model is used to capture semantics and obtain word vectors that can reflect the semantic characteristics of the domain. These word vectors not only contain the co-occurrence relationship between words, but also imply the semantic information of the words in the specific context.

[0048] Through similarity evaluation, we can ensure that the generated word vectors can accurately reflect the semantic features in a specific field. High-quality word vectors can better capture the vocabulary relationship and semantic information in the field, thereby improving the performance of subsequent event extraction models. Similarity evaluation can screen out high-quality word vectors based on the set evaluation indicators and thresholds. For example, using cosine similarity evaluation, word vectors with a cosine similarity higher than 0.8 are considered to be of high quality. However, in actual applications, it is necessary to select appropriate evaluation methods and indicators based on specific application scenarios and task requirements, and adjust and optimize according to specific circumstances.

[0049] Domain word vectors are used in vocabulary-level embedding representations. During training, domain word vectors can be used as initial embedding layer parameters to help the model converge faster and improve model performance. Especially when processing text in a specific field, domain word vectors can provide more accurate initial representations and accelerate the model's learning process.

[0050] In the above technical solution, in the data set construction module, specific event data is obtained and preprocessed. The preprocessing method includes: removing irrelevant information, stop words, noise, and word segmentation. The preprocessed data is converted into a sequence annotation format, such as BIO (Begin, Inside, Outside) format. Among them, "B-" means that the word is the beginning of an entity or argument; "I-" means that the word is part of an entity or argument, but not the beginning; "O-" means that the word does not belong to any entity or argument.

[0051] In the data set construction module, the purity and accuracy of the data are ensured by obtaining data of specific events and performing preprocessing, including removing irrelevant information, stop words and noise, and segmenting words. The preprocessed data is converted into a sequence annotation format, which is convenient for model understanding and learning, improves the accuracy and efficiency of the task, simplifies the subsequent data processing process, and enhances the effect and generalization ability of model training. In the above technical solution, in the event extraction model construction module, the feature extraction layer includes using the LERT sub-model and the Soft-lexicon sub-model to convert the specific field text sequence data into character-level embedding representations. char,i and the vocabulary-level embedding representation e word,i , where specifically:

[0052] The above LERT sub-model uses a multi-layer Transformer architecture as its core encoder, and uses the LERT sub-model to convert domain-specific text sequence data into character-level embedding representations. char,i The specific operation is: for the input sentence s=(c1,c2,…,c k ), unique tags [CLS] and [SEP] are added to indicate the beginning and end of the sentence, and each token is concatenated into an embedded representation through character embedding, paragraph embedding, and position embedding, that is:

[0053]

[0054] Among them, E ci Represents the character c i The complete embedding representation, E token stands for character embedding, E seg represents paragraph embedding, E pos Represents positional embedding.

[0055] Then sentence s=(c1,c2,…,c k ) is converted to a vector sequence Input into the bidirectional Transformer to obtain features. The Transformer's self-attention mechanism captures the context information of each token and generates a sequence vector e rich in semantic features. char,i ={x1,x2,…,x k}.

[0056] Utilize the Soft-lexicon sub-model to convert domain-specific text sequence data into vocabulary-level embedding representations. word，i The specific process is: for the input text s=(c1,c2,…,c k ), when processing each character c i When a character matches a word in the dictionary, it is compared with the word in the dictionary. When the character matches a word in the dictionary, the character is classified into the corresponding set. i The four sets are as follows:

[0057]

[0058]

[0059]

[0060]

[0061] Among them, L represents the dictionary, w m,n Indicates that c m Start with c n The word in the dictionary that ends, B(c i ) represents c i The starting word set, M(c i ) represents c i is the word set with the middle word, E(c i ) represents c i The word set ending with S(c i ) means only contains c i A set of single characters. If no word matching the {B,M,E,S} structure is found in the dictionary, the set is empty.

[0062] After constructing the character word set, these word sets are converted into vector representations with fixed dimensions by counting the frequency of occurrence of each word in the word set, as shown in the following formula:

[0063]

[0064]

[0065] Among them, B, M, E, and S represent a word set respectively, and v s(S), v s (M), v s (E) v s (S) represents the weights of word sets B, M, E, and S respectively, z(w) represents the frequency of word w in the statistical word set, e w (w) represents the vector corresponding to word w in the domain word vector lookup table, Z represents the normalization factor, which is used to ensure that the weighted vocabulary information vector has an appropriate scale, and e s (B,M,E,S) is the total vector representation of the four vocabulary sets B,M,E,S, e word,i ←{s,e s (B,M,E,S)}e word,i Represents the character vector after integrating vocabulary information, that is, vocabulary-level embedding.

[0066] The above process assigns a vector that integrates vocabulary information to each character. In this way, each character in the input text sequence not only retains its original word vector, but also integrates the vocabulary information corresponding to it, realizing a one-to-one correspondence between the character and its vocabulary information.

[0067] The use of the Soft-lexicon sub-model mentioned above is a simple and efficient way to incorporate vocabulary information into characters. This method only requires minor adjustments at the character representation layer to incorporate vocabulary information into the character representation, without the need to design a complex sequence modeling structure, which effectively simplifies the model design while enhancing the model's ability to process vocabulary information. At the feature extraction layer, each character is mapped to a high-dimensional vector, and a soft dictionary feature is constructed, which is then merged with the representation of each character. This method effectively makes up for the shortcomings of models based only on character information in utilizing vocabulary information, and while maintaining a high inference speed, it also has better performance and can be more easily combined with pre-trained models such as BERT.

[0068] In the above technical solution, in the event extraction model construction module, the feature fusion layer fuses the character-level embedding representation and the word-level character representation, and the fused vector is represented as e fusion,i :

[0069]

[0070] in, Represents a vector concatenation operation.

[0071] The fusion layer combines the character-level embedding representation and the word-level embedding representation to make full use of semantic information at two different granularities.

[0072] The input text sequence obtains character-level embedding representation through the LERT sub-model, and adjusts and optimizes the vocabulary-level embedding representation through the Soft-lexicon sub-model. k ), the LERT sub-model converts each character c i Converted to an embedding vector e char,i Similarly, the Soft-lexiocn sub-model converts each word w i Converted to an embedding vector e word,i The fused representation is fusion It can be expressed as:

[0073]

[0074] in, Represents a vector concatenation operation.

[0075] The fusion layer is a bridge connecting character representation and vocabulary representation, providing the model with more comprehensive and rich input features.

[0076] In the above technical solution, in the event extraction model construction module, the output decoding layer is structured as an encoder, a decoder, and a CRF layer, specifically:

[0077] The encoder is used to map the fusion vector provided by the feature fusion layer into a continuous semantic vector. The encoder includes a multi-head self-attention sublayer, a residual connection and normalization layer, and a feed-forward fully connected network sublayer.

[0078] The decoder is used to gradually generate the label sequence of the output characters according to the output of the encoder. The decoder includes a masked multi-head self-attention layer, an encoder-decoder self-attention layer, a feed-forward fully connected network sublayer, a linear layer, and a Softmax function.

[0079] The CRF layer is used to learn the transition probability between labels based on the label sequence of the characters, obtain the probability of each label sequence combination, and take the label sequence combination with the highest probability as the final output.

[0080] The Transformer model constitutes the core component of the output decoding layer. It is responsible for deeply understanding the bidirectional context of the input sequence and capturing the long-distance dependencies therein. The Transformer model achieves efficient processing of event feature extraction tasks through its unique encoder-decoder structure; the CRF layer can more accurately capture the dependencies between labels by learning the transition probabilities between labels, thereby improving the accuracy of annotation.

[0081] The output decoding layer is based on the Transformer model, and a CRF layer is introduced after the decoder of the Transformer model to further optimize the predicted label sequence.

[0082] The Transformer model includes an encoder and a decoder. The specific construction process is as follows:

[0083] Encoder: Each layer contains a multi-head self-attention sublayer, which captures the contextual information in the input sequence by calculating the attention weights between the query, key, and value. The formula is as follows:

[0084]

[0085] Among them, Q, K, and V represent query, key, and value respectively. k is the dimension of the key vector, Used to scale the dot product to prevent the vanishing gradient problem. In this way, the model is able to focus on multiple positions in the input sequence in parallel, thereby gaining a more comprehensive understanding of the text content.

[0086] In order to enhance the stability and training efficiency of the model, the output of each sub-layer is added to its input (i.e. residual connection), and then the layer is normalized. The formula is as follows:

[0087] LayerNorm(x+Sublayer(x));

[0088] Here, x is the input of the sublayer, Sublayer(x) is the output of the sublayer, and the addition operation represents the residual connection. This step ensures that the model can maintain a stable gradient flow during training and accelerates convergence.

[0089] After the multi-head self-attention sublayer, the encoder also contains a feed-forward fully connected network sublayer. This sublayer consists of two linear layers, with a nonlinear transformation performed through the ReLU activation function in the middle. The formula is as follows:

[0090] FFN(x)=max(0,xW1+b1)W2+b2;

[0091] Among them, W1 and W2 are weight matrices, and b1 and b2 are bias terms. This step further extracts high-dimensional features from the input sequence and provides richer semantic information for the subsequent decoder.

[0092] The encoder part of the Transformer is responsible for further mapping the rich input feature representation provided by the feature fusion layer into continuous semantic vectors. These semantic vectors not only contain character-level and word-level embedding information, but also capture the dependencies between different positions in the input sequence through a multi-head self-attention mechanism.

[0093] Decoder: The decoder part gradually generates an output sequence, i.e., event elements, based on the output of the encoder. Each layer of the decoder consists of three sublayers: a masked multi-head self-attention layer, an encoder-decoder attention layer, and a feed-forward fully connected network.

[0094] Masked multi-head self-attention layer: This sublayer ensures that the model does not depend on future outputs when generating a certain output, thus avoiding leakage of future information. This is crucial for sequence generation tasks because it ensures that the generated sequence is logical and grammatical.

[0095] Encoder-Decoder Attention Layer: This sublayer allows the decoder to focus on the output of the encoder, thereby capturing the dependencies between the input and output sequences. By calculating the attention weights between the encoder output and the current state of the decoder, the model is able to more accurately generate event features corresponding to the input sequence.

[0096] Feedforward fully connected network: The feedforward fully connected network sublayer in the decoder is similar to that in the encoder and is used to further extract and decode high-dimensional features in the input sequence.

[0097] Finally, the output of the last layer of the decoder is converted into a predicted probability distribution through a linear layer and a softmax function. These probability distributions correspond to event elements in the text, and the model generates the final output sequence based on these probability distributions.

[0098] The Transformer decoder part gradually decodes the event elements corresponding to the input sequence.

[0099] However, relying solely on the output of the Transformer decoder cannot fully capture the dependencies between labels, especially in sequence labeling tasks. Therefore, after the last layer of the decoder, we introduce a CRF layer to further optimize the predicted label sequence.

[0100] The Transformer model is used to combine the contextual information around each position in the text sequence to enhance the understanding of the current position. While maintaining sensitivity to long-distance dependencies, the Transformer fully considers the elements or contextual information near each position in the sequence, that is, the output vector contains both the information of the current position itself and the information of other positions in the sequence, thereby realizing the integration of local features and providing a richer and more accurate representation.

[0101] The construction process of the CRF layer is:

[0102] The CRF layer can more accurately capture the dependencies between labels by learning the transition probabilities between labels, thereby improving the accuracy of annotation. During the decoding process, the CRF layer receives the output of the last layer of the decoder as input and calculates the probability of each possible label sequence combination. Then, the label sequence combination with the highest probability is selected as the final output.

[0103] Specifically, the CRF layer is optimized through the following steps:

[0104] Step 1. Calculate transition probabilities: The CRF layer first learns the transition probabilities between labels. These probabilities represent the likelihood of transferring from one label to another.

[0105] Step 2: Calculate path probabilities: The CRF layer then calculates the probabilities of all possible combinations of label sequences (i.e., paths). This is achieved by multiplying the output of the last layer of the decoder with the emission probability of the label (i.e., the probability that the decoder output corresponds to each label) and the transition probability between labels.

[0106] Step 3: Select the optimal path: The CRF layer selects the label sequence combination with the highest probability as the final output. This optimal path is the event feature sequence predicted by the model.

[0107] In the above technical solution, in the training module, the annotated specific domain text sequence data set includes a training set, a validation set and a test set. The training module also includes a validation and testing process. The specific construction method of the training module is: using the domain word vector as the initialization setting for model training, using the training set to train the training event extraction model, using the validation set and the test set to verify and test the trained model, outputting the predicted sequence, that is, the result of event extraction, and obtaining the precision, recall rate and F1 value.

[0108] Using domain word vectors as the initial setting for model training, i.e. the initial representation, can help the model converge faster and improve the performance of the model. Especially when processing text in a specific field, domain word vectors can provide more accurate initial representation and accelerate the learning process of the model.

[0109] The training set is used to optimize the model parameters to ensure that it learns the correct pattern; the validation set is used to tune the hyperparameters and prevent overfitting to ensure the generalization ability of the model. Finally, the test set evaluates the actual performance of the model and outputs the prediction sequence, i.e. the result of event extraction. By calculating indicators such as precision, recall, and F1 value, we can comprehensively measure the performance of the model to ensure its reliability and effectiveness in practical applications. This rigorous training and evaluation process not only improves the quality of the model, but also lays a solid foundation for subsequent application deployment.

[0110] The process of training the model on the training set is:

[0111] 1) Input the dataset D, word vector file gaz_file and pre-trained model file, set the model parameters and the maximum number of iterations Epoch;

[0112] 2) Initialize model parameters;

[0113] 3) Load the word vector file gaz_file;

[0114] 4) Load the pre-trained model file;

[0115] 5) Start training. For each iteration, read the training set D_train, divide it into multiple batches, and perform the following operations on each batch:

[0116] Convert text sequences into tensor format for model input;

[0117] Forward propagation calculates loss loss;

[0118] Back propagate and update parameters;

[0119] Until the iteration is completed.

[0120] In order to better demonstrate the effectiveness and practicality of the event extraction system based on the pre-training model and word meaning enhancement, as shown in Table 1, the MNEE dataset is used for experimental verification. The MNEE dataset is a military news event dataset constructed by the National University of Defense Technology based on public military news content, which contains 8 event types and 10 types of event arguments.

[0121] Table 1 Data display of the model of the present invention compared with other models

[0122]

[0123] Taking LSTM-CRF as the basic model, the three models of Lattice LSTM, LR-CNN, and Flat-Lattice Transformer are all sequence labeling methods that perform semantic enhancement based on vocabulary information. The core idea is to design a dynamic framework that is compatible with vocabulary input and enhance the model's ability to process Chinese text by introducing vocabulary-level information. Vocabulary information provides the model with richer contextual understanding and helps to more accurately identify event elements. The experimental results show that the effect of introducing vocabulary information has been improved to a certain extent compared to the basic LSTM-CRF.

[0124] Lattice LSTM integrates character-level and vocabulary-level information into the LSTM network by building a Lattice structure, improving the ability to recognize event elements. Experimental results show that Lattice LSTM exceeds LSTM-CRF in both precision and recall, and the F1 value is improved by 7.95%, indicating that the introduction of vocabulary information can significantly enhance the model's ability to recognize event trigger words.

[0125] LR-CNN improves the performance of event extraction by using CNN-based character feature encoding and vocabulary information fusion. The characteristic of the FLAT method is that the boundary information of the Lattice vocabulary is added to the model by designing the position vector.

[0126] BERT-BiLSTM-CRF, ALBERT-BiLSTM-CRF, and RoBERTa-BiLSTM-CRF all use pre-trained language models for event extraction. The experimental results show that the introduction of pre-trained models improves the extraction effect to varying degrees, and is superior to the previous models in all indicators. This shows that pre-trained models can provide rich semantic representations and enhance the model's ability to understand text.

[0127] The model of the present invention integrates the deep semantic understanding ability of the pre-trained language model and the semantic enhancement effect of domain vocabulary information, and performs information fusion at both the character level and the vocabulary level. This multi-level information processing helps the model capture more detailed semantic differences and effectively improves the model's semantic understanding ability of texts in specific fields.

[0128] Embodiment 2:

[0129] An event extraction method based on a pre-trained model and word meaning enhancement includes the following steps:

[0130] Step 1: Use the public text data of specific field events to build a specific field corpus, and obtain the field word vector by training and evaluating the specific field corpus;

[0131] Step 2: Obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset;

[0132] Step 3: Using the feature extraction layer, feature fusion layer and output decoding layer as the model framework, construct an event extraction model;

[0133] Step 4: Input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled specific domain text sequence dataset to obtain a trained event extraction model.

[0134] Step 5: Based on the specific domain text sequence dataset to be detected, use the trained event extraction model to calculate the probability of detecting each label sequence combination based on character combination in the specific domain text sequence dataset, and obtain the label sequence combination with the highest probability in the specific domain text sequence dataset.

[0135] In step 1, by collecting and organizing public text materials in a specific field, a high-quality domain corpus is constructed, and a dedicated domain word vector is trained. This step ensures that the model can understand the terminology and expressions of a specific industry and improves its ability to understand professional texts. The domain word vector captures the contextual relationship of the vocabulary in the field, providing a more accurate representation basis for subsequent feature extraction, making the model more efficient in processing domain-related tasks.

[0136] In step 2, the collected domain-specific text is converted into a dataset in a sequence annotation format, which can standardize the data structure and provide clear target labels for model learning. This format conversion allows each word or phrase to be associated with a specific event element, which facilitates supervised learning of the model. The converted dataset can effectively guide the model to identify and classify key information in the text, improving the accuracy and efficiency of event extraction. In step 3, the feature extraction layer combines the LERT sub-model and the Soft-lexicon sub-model to process the text sequence at the character level and the vocabulary level respectively to obtain a deep language feature representation. The LERT sub-model focuses on character-level details, which helps to capture linguistic features; while Soft-lexicon uses pre-trained domain word vectors to enhance semantic understanding at the vocabulary level. The combination of the two provides a rich and complementary source of information for subsequent fusion, greatly improving the model's ability to parse complex language phenomena.

[0137] The feature fusion layer generates a fusion vector by concatenating the embedding representations at the character level and the word level, integrating the feature representations of two different granularities, and enhancing the representation ability of the model. The fused vector contains both detailed character information and high-level lexical meanings, providing a more comprehensive and in-depth understanding for the output decoding layer. This method effectively solves the limitations that may be brought about by single-level features and promotes a more accurate grasp of the text content by the model.

[0138] The output decoding layer uses the Transformer model to capture the long-distance dependencies in the input fusion vector, while integrating local features to ensure the consistency and logical coherence of the output sequence. The CRF (conditional random field) layer is introduced to calculate the probability distribution of all possible label sequences, and finally the sequence with the highest probability is selected as the prediction result. The above operation can not only improve the adaptability of the model to complex syntactic structures, but also enhance the precise positioning of event element boundaries, significantly improving the accuracy and reliability of event extraction.

[0139] In step 4, the domain word vector is used as the initial representation for model training, and the event extraction model is trained using a known data set, which significantly improves the professionalism and accuracy of the model. The domain word vector pre-captures the semantic features of a specific industry and provides a rich initial weight for the model, so that the model has the ability to understand professional terms and expressions from the beginning. During the training process, these initial representations are combined with the real labels in the data set to help the model learn domain-specific patterns and rules more efficiently. Not only does it speed up the convergence speed, but it also enhances the model's understanding and generalization ability of complex texts, ensuring that it can accurately identify and extract event elements in practical applications, thereby providing more reliable and valuable analysis results. In step 5, after inputting the text sequence to be detected, the trained event extraction model is used to analyze it, predict and output the event elements contained therein. This step puts all the preparatory work and model training results into practice, realizing the transformation from theory to application. In this way, key event information in documents can be quickly located and extracted, providing users with valuable insights, and is widely applicable to various specific fields.

[0140] Embodiment 3:

[0141] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0142] Embodiment 4:

[0143] A computer program product comprises a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0144] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

[0149] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. An event extraction system based on pre-training model and word meaning enhancement, characterized in that: include: The domain word vector acquisition module is used to construct a domain-specific corpus using the text materials disclosed by events in a specific domain, and obtain domain word vectors by training and evaluating the domain-specific corpus; The data set construction module is used to obtain a specific domain data set, convert the specific domain data set format into a sequence annotation format, and obtain an annotated specific domain text sequence data set; The event extraction model construction module is used to construct an event extraction model with the feature extraction layer, feature fusion layer and output decoding layer as the model framework; The training module is used to input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled specific domain text sequence data set to obtain a trained event extraction model. The event extraction module is used to calculate the probability of detecting each label sequence combination based on character combination in the specific field text sequence dataset based on the trained event extraction model, and obtain the label sequence combination with the highest probability in the specific field text sequence dataset.

2. The event extraction system based on pre-training model and word meaning enhancement according to claim 1, characterized in that: Taking the feature extraction layer, feature fusion layer and output decoding layer as the model framework, the specific method of constructing the event extraction model is as follows: The feature extraction layer processes the labeled domain-specific text sequence dataset at the character level and the word level through the LERT sub-model and the Soft-lexicon sub-model in the event extraction model to obtain the character-level embedding representation and the word-level embedding representation; The feature fusion layer performs a fusion and splicing operation on the character-level embedding representation and the word-level embedding representation to obtain a fused vector. The output decoding layer captures the long-distance dependencies of characters in the fusion vector, integrates local features, obtains the label sequence of each character, and uses the CRF layer to calculate the probability of each label sequence combination based on character combination in the labeled specific domain text sequence dataset to form an event extraction model.

3. The event extraction system based on pre-training model and word meaning enhancement according to claim 1, characterized in that: In the domain word vector acquisition module, the specific process of obtaining the domain word vector is as follows: Use the public text materials of events in a specific field to build a field corpus, clean and denoise the field corpus, remove irrelevant information and stop words, and identify field professional terms and common words; The Word2Vec model is used to train the cleaned corpus to obtain word vectors that can reflect the semantic characteristics of the domain; The quality of word vectors is evaluated by similarity evaluation method.

4. The event extraction system based on pre-training model and word meaning enhancement according to claim 1, characterized in that: In the event extraction model construction module, the feature extraction layer includes using the LERT sub-model and the Soft-lexicon sub-model to convert the domain-specific text sequence data into character-level embedding representations. char,i and the vocabulary-level embedding representation e word,i , where the LERT sub-model is used to convert into a character-level embedding representation e char,i The specific method is: The acquired domain-specific text sequence data s=(c1,c2,…,c k ) mark the start and end, where c1, c2, …, c k Represents k characters; Each character is concatenated into an embedding representation through character embedding, paragraph embedding and position embedding, expressed as: Among them, E ci Represents the character c i The complete embedding representation, E token stands for character embedding, E seg represents paragraph embedding, E pos represents positional embedding; Convert domain-specific text sequence data into vector sequences based on the concatenated embedding representation Use Transformer to obtain features and get character-level embedding representation e chari ={x1,x2,…,x k }; The domain word vector is used as the initial representation trained in the Soft-lexicon sub-model. The trained Soft-lexicon sub-model obtains the vocabulary-level embedding representation e word,i , specifically: The specific domain text sequence data s=(c1,c2,…,c k ) is compared with the words in the dictionary and divided accordingly to construct a character word set. The specific division method is: Among them, L represents the dictionary, w m,n Indicates that c m Start with c n The word in the dictionary that ends, B(c i ) represents c i The starting word set, M(c i ) represents c i is the word set with the middle word, E(c i ) represents c i The word set ending with S(c i ) means only contains c i A set of single characters. If no word that conforms to the {B,M,E,S} structure is found in the dictionary, the set is an empty set. After constructing the character word set, the frequency of occurrence of each word in the character word set is counted and converted into a vector representation with a fixed dimension to obtain the vocabulary-level embedding e word,i , specifically expressed as: Among them, B, M, E, and S represent a word set respectively, and v s (S), v s (M), v s (E) v s (S) represents the weights of word sets B, M, E, and S respectively, z(w) represents the frequency of word w in the statistical word set, e w (w) represents the vector corresponding to word w in the domain word vector lookup table, Z represents the normalization factor, which is used to ensure that the weighted vocabulary information vector has an appropriate scale, and e s (B,M,E,S) is the total vector representation of the four vocabulary sets B,M,E,S, e word,i ←{s,e s (B,M,E,S)},e word,i Represents the character vector after integrating vocabulary information, that is, vocabulary-level embedding.

5. The event extraction system based on pre-training model and word meaning enhancement according to claim 4, characterized in that: In the event extraction model construction module, the feature fusion layer fuses the character-level embedding representation and the word-level character representation, and the fused vector is represented as e fusion,i : in, Represents a vector concatenation operation.

6. The event extraction system based on pre-training model and word meaning enhancement according to claim 1, characterized in that: In the event extraction model construction module, the output decoding layer is composed of an encoder, a decoder, and a CRF layer, specifically: The encoder is used to map the fusion vector provided by the feature fusion layer into a continuous semantic vector. The encoder includes a multi-head self-attention sublayer, a residual connection and normalization layer, and a feed-forward fully connected network sublayer. The decoder is used to gradually generate the label sequence of the output characters according to the output of the encoder. The decoder includes a masked multi-head self-attention layer, an encoder-decoder self-attention layer, a feed-forward fully connected network sublayer, a linear layer, and a Softmax function. The CRF layer is used to learn the transition probability between labels based on the label sequence of the characters, obtain the probability of each label sequence combination, and take the label sequence combination with the highest probability as the final output.

7. According to the event extraction system based on pre-training model and word meaning enhancement described in claim 1, in the training module, the annotated specific domain text sequence data set includes a training set, a verification set and a test set, and the training module also includes a verification and testing process. The specific construction method of the training module is: using the domain word vector as the initialization setting for model training, using the training set to train the training event extraction model, using the verification set and the test set to verify and test the trained model, outputting the predicted sequence, that is, the result of event extraction, and obtaining the precision, recall rate and F1 value.

8. An event extraction method based on a pre-trained model and word meaning enhancement, characterized in that: The steps include: The domain word vector acquisition module is used to construct a domain-specific corpus using the text materials disclosed by events in a specific domain, and obtain domain word vectors by training and evaluating the domain-specific corpus; Obtain a specific domain dataset, convert the specific domain dataset format into a sequence annotation format, and obtain an annotated specific domain text sequence dataset; The event extraction model is constructed with the feature extraction layer, feature fusion layer and output decoding layer as the model framework; Input the domain word vector into the Soft-lexicon sub-model in the event extraction model for initialization. After the Soft-lexicon sub-model is initialized, the event extraction model is trained using a labeled domain-specific text sequence dataset to obtain a trained event extraction model. According to the specific domain text sequence dataset to be detected, the trained event extraction model is used to calculate the probability of detecting each label sequence combination based on character combination in the specific domain text sequence dataset, and obtain the label sequence combination with the highest probability in the specific domain text sequence dataset.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in claim 8 are implemented.

Images