MoE-fused cascade enterprise news event extraction method

Abstract

The invention discloses a cascade enterprise news event extraction method fusing MoE, and the method comprises the following steps: 1) obtaining enterprise news event texts from all financial platforms and news websites, and carrying out the preprocessing of the data; 2) performing parameter adjustment on the large language model, and adjusting a parameter adapter and a classification layer related to event detection of the large language model; 3) introducing a MoE mechanism into the adjusted large model to enhance the effect of model event detection, and performing event detection on the news text through a model fused with MoE; and 4) dynamically constructing a semantic constraint prompt template based on an event detection result, guiding the model to focus on a text fragment related to a target role, and completing extraction of news elements.

Description

[0001] Technical space

[0002] The present invention belongs to the fields of natural language processing and deep learning, and in particular, proposes a cascaded enterprise news event extraction method integrating MoE. Background technology

[0003] Enterprise news, as the key information directly reflecting the operation status and strategic trends of enterprises, has formed a three-level diffusion effect of "enterprise - industrial chain - industry" in its value transmission mechanism. However, the unstructured characteristics of a large amount of news lead to problems such as low efficiency and subjective deviation in manual extraction of key events, making it difficult to support the accurate tracking of enterprise dynamics and the quantitative analysis of industrial chain trends. Therefore, using computer technology to structurally process the event information in news texts has important practical significance. This method can not only improve the information extraction efficiency but also lay a foundation for subsequent data analysis, drive the intelligence of enterprise decision-making, and assist in industrial trend analysis and risk warning, etc.

[0004] Event Extraction (EE), as a key task in the field of information extraction, aims to identify specific types of events and present the elements playing established roles in the events in a structured form. The event extraction task usually covers two core links: Event Detection (ED) and Event Argument Extraction (EAE). Event Detection determines the event category by analyzing the text content, and Event Argument Extraction extracts various participating elements related to the event from the document based on the determined event type.

[0005] The event extraction task is divided into sentence-level event extraction and document-level event extraction according to different research objects. Given that enterprise news texts are generally long in length and a single news often contains multiple independent events, the present invention focuses on document-level event extraction. Compared with sentence-level event extraction, document-level event extraction faces a more complex task scenario, and its core challenges are multi-event extraction and argument dispersion, etc. First, news texts may contain multiple events, and the arguments of different events may be nested. Ensuring the accurate identification of event types is the basis for event argument extraction. Second, since the different arguments of an event may be distributed in multiple sentences of the document, it is often difficult to obtain complete event information if event extraction is only based on a single sentence. In addition, there is also the phenomenon of argument overlap in event arguments, and an argument may be an argument of multiple events.

[0006] The work done on event extraction using large language models is very limited. AugGPT is a recent method that generates data for downstream model training by restating each sentence in the training samples into multiple conceptually similar but semantically different samples, overcoming the deficiencies of current text data augmentation methods in label correctness and data diversity. Whitehouse C et al. enhanced the dataset using multiple large language models to detect event trigger words and then predict event types, and adjusted the large language model to improve the performance of model event extraction. Although these two methods have achieved good results, there are still the following deficiencies:

[0007] (1) Large language models can obtain better event detection effects by understanding semantics through context semantic modeling on long texts. However, due to the uncontrollability of the output elements of the generative model, there is a problem of fuzzy output boundaries. Moreover, in document-level event extraction, it is difficult to directly extract information due to the parallelism of multiple events and the dispersion of arguments in enterprise news texts.

[0008] (2) Although large language models enhance the context modeling ability through the pre-training mechanism, they still face problems such as uncontrollable generation and high computational resource consumption. Although existing large language models enhance the context modeling ability through the pre-training mechanism, their hundreds of billions of parameters lead to too high training and inference costs, making it difficult to meet the real-time needs of enterprises. Summary of the Invention

[0009] Object of the Invention. In view of the above defects or problems, a cascaded enterprise news event extraction method integrating MoE is provided. Driven by the event detection results, the model dynamically generates a prompt template with role constraints through a predefined event template library, transforms the argument extraction task into a constrained text generation problem, and realizes the extraction of event arguments.

[0010] Technical Solution. To solve the above technical problems, the present invention proposes a cascaded enterprise news event extraction method integrating MoE, which includes the following steps:

[0011] Step 1) Obtain enterprise news event texts from various financial platforms and news websites, and preprocess the data;

[0012] Step 2) Adjust the parameters of the large language model, and adjust its parameter adapter and classification layer related to event detection;

[0013] Step 3) Introduce the MoE mechanism into the adjusted large model to enhance the event detection effect of the model, and perform event detection on news texts through the model integrating MoE;

[0014] Step 4) Dynamically construct a semantic-constrained prompt template based on the event detection results, guide the model to focus on the text segments related to the target role, and complete the extraction of news elements.

[0015] Further, the specific method of step 1) is as follows:

[0016] Step 1.1) Select financial platforms and news websites as data sources, and regularly obtain news data; use keyword search or topic crawling methods to obtain news texts related to enterprises, and set the data update frequency to ensure obtaining the latest enterprise news events;

[0017] Step 1.2) Preprocess the data, filter advertisements, promotions and non-news content through regular expressions and remove duplicates, perform Unicode normalization on the text, unify the encoding format and remove abnormal characters and invalid punctuation to improve the text quality and event relevance.

[0018] Further, the specific method of step 2) is as follows:

[0019] Step 2.1) Freeze the weights of the pre-trained model, independently train a decomposition matrix, combine this matrix with the weights of the pre-trained model to achieve model adjustment. For the original weight matrix W, its change ΔW is approximately expressed as the product of two matrices A and B, as shown in formula (1):

[0020] ΔW≈A*B T (1)

[0021] Step 2.2) Introduce two matrices A and B into the weight matrix W O of the pre-trained model, and its adjustment process is as shown in formula (2):

[0022]

[0023] In formula (2), A is initialized with a random Gaussian distribution, and B is initialized with 0. At the beginning of training, BA is 0, and a scaling factor is introduced for scaling parameters to further optimize the training process of the model. Among them, ɑ represents the scaling amplitude coefficient, which is used to control the overall size of the update amount; r represents the rank of the decomposition matrix.

[0024] Further, the specific method of step 3) is as follows:

[0025] Step 3.1) Regard the event detection task as a multi-label classification task. Given the input text sequence X = [x1, x2,..., x n , the model needs to predict the activation status of the event type set Y = {y1, y2,..., y k}, where k represents the total number of event types, and each y i ∈{0, 1};

[0026] Step 3.2) Change the language modeling layer of the model to a classification layer, and add a MoE module in front of the classification layer. This module is jointly composed of multiple expert models and a gating model. The gating model assigns corresponding weights to each expert model. For the input sample, assuming the output of the \(i\)-th expert model is \(o\). i , and the corresponding true label is \(t\), then the loss function of this model is shown in Equation (3):

[0027]

[0028] In Equation (3), \(p\). i represents the weight assigned by the gating model to the \(i\)-th expert model. The weight \(p\). i is placed outside the loss function, enabling each expert model to independently calculate its loss value and making different expert models compete with each other.

[0029] Step 3.3) Each expert model performs classification prediction on the input data and outputs the probability distribution of each category; the gating model quantifies the importance of each expert model to the final classification result through the assigned weights.

[0030] Step 3.4) The weights calculated by the gating network are used to weighted aggregate the outputs of each expert model. The prediction result of each expert model is multiplied by its corresponding weight, and the sum of these weighted results is calculated to obtain the final prediction output. The output form of MoE is shown in Equations (4) and (5):

[0031]

[0032] In Equation (4), \(p\). t represents the weight of the \(t\)-th expert model, represents the weighted sum of the prediction results of each model, \(N\) represents the number of expert models participating in the weighting. To ensure the rationality of the probability distribution, it is required that the sum of the weights of all models is 1, so as to ensure that the sum of the final output probabilities is still 1.

[0033] Step 3.5) The fully connected layer of the model classification layer maps the hidden state to the dimension of the number of event categories, and uses a new weight matrix and bias to perform a linear transformation, as shown in Equation (6):

[0034] \(z\). cls = \(W\). cls + \(b\). cls (6)

[0035] The activation function of the classification layer is the sigmoid activation function. This activation function independently maps the value of each output neuron and converts it into a probability value between [0, 1]. Each value represents the probability of the corresponding event category, and the probabilities of different event categories are independent of each other, as shown in formula (7), where z represents the input real value, which is calculated by formula (6).

[0036]

[0037] Step 3.6) Use the fine-tuned event detection model with fused MoE for event detection in news texts and finally output the event types.

[0038] Furthermore, the specific method of step 4) is as follows:

[0039] Step 4.1) After completing the event type detection, the model constructs the input sequence through a cascaded architecture to achieve structured event argument extraction. The predefined event template library is shown in formula (8):

[0040]

[0041] In formula (8), e i represents the event type, is the corresponding role set, and the parameter m represents the number of argument roles to be extracted under this event. Through the dynamic prompt template based on the event template, the event detection and argument extraction are decoupled into two-level pipeline tasks;

[0042] Step 4.2) For the detected event type e i ∈Y, the system retrieves its role definition R t from the template library, constructs a prompt template with clear semantic constraints, and uses the constructed prompt template as the input of the event argument extraction module to achieve the extraction of event arguments.

[0043] Beneficial effects. Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0044] The present invention obtains news events from the open knowledge network and uses the method of cascaded enterprise news event extraction with fused MoE to extract news event elements. It quickly captures the key elements in news events and the trend changes in enterprise development, providing a basis for enterprise public opinion monitoring and market trend analysis. Description of the Drawings

[0045] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below.

[0046] Figure 1 is the flowchart of the embodiment of the present invention;

[0047] Figure 2 It is a display diagram of the input prompt template for event argument extraction in an embodiment of the present invention.

[0048] Specific implementation

[0049] Such as Figure 1 As shown, the present invention proposes a cascaded enterprise news event extraction method integrating MoE, and the method includes the following:

[0050] Step 1) Obtain enterprise news event texts from various financial platforms and news websites, and preprocess the data;

[0051] Step 2) Adjust the parameters of the large language model, and adjust its parameter adapters and classification layers related to event detection;

[0052] Step 3) Introduce the MoE mechanism into the adjusted large model to enhance the effect of model event detection, and perform event detection on news texts through the model integrating MoE;

[0053] Step 4) Dynamically construct a semantic constraint prompt template based on the event detection results, guide the model to focus on text segments related to the target role, and finally complete the extraction of news elements.

[0054] Furthermore, the specific method of step 1) is as follows:

[0055] Step 1.1) Select financial platforms and mainstream news websites as data sources, and regularly obtain news data; use keyword search or topic crawling methods to obtain enterprise-related news texts, and set the data update frequency to ensure the acquisition of the latest enterprise news events;

[0056] Step 1.2) Preprocess the data, filter advertisements, promotions and non-news content through regular expressions and remove duplicates, perform Unicode normalization processing on the text, unify the encoding format and remove abnormal characters and invalid punctuation to improve the text quality and event relevance.

[0057] Furthermore, the specific method of step 2) is as follows:

[0058] Step 2.1) Freeze the weights of the pre-trained model, independently train a decomposition matrix, combine the matrix with the weights of the pre-trained model to achieve model adjustment. For the original weight matrix W, its change ΔW is approximately expressed as the product of two matrices A and B, as shown in formula (1):

[0059] ΔW≈A*B T (1)

[0060] Step 2.2) The weight matrix Wo of the pre-trained model OTwo matrices A and B are introduced, and the adjustment process is shown in Equation (2):

[0061]

[0062] In Equation (2), A is initialized with random Gaussian distribution, and B is initialized with 0. At the beginning of training, BA is 0, and a scaling factor is introduced to perform scaling parameters to further optimize the training process of the model. Here, ɑ represents the scaling amplitude coefficient, which is used to control the overall size of the update amount; r represents the rank of the decomposition matrix.

[0063] Furthermore, the specific method of step 3) is as follows:

[0064] Step 3.1) Consider the event detection task as a multi-label classification task. Given the input text sequence X = [x1, x2,..., x n , the model needs to predict the activation status of the event type set Y = {y1, y2,..., y k}, where k represents the total number of event types, and each y i ∈{0, 1};

[0065] Step 3.2) Change the language modeling layer of the model to a classification layer, and add a MoE module in front of the classification layer. This module is composed of multiple expert models and a gating model. The gating model assigns corresponding weights to each expert model. For the input sample, assume the output of the i-th expert model is o i , and the corresponding true label is t. Then the loss function of this model is shown in Equation (3):

[0066]

[0067] In Equation (3), p i represents the weight assigned by the gating model to the i-th expert model. The weight p i is placed outside the loss function, enabling each expert model to independently calculate its loss value and allowing different expert models to compete.

[0068] Step 3.3) Each expert model performs classification prediction on the input data and outputs the probability distribution of each category; the gating model quantifies the importance of each expert model to the final classification result through the assigned weights;

[0069] Step 3.4) The weights calculated by the gating network are used to weighted aggregate the outputs of each expert model. The prediction result of each expert model is multiplied by its corresponding weight, and the sum of these weighted results is calculated to obtain the final prediction output. The output form of MoE is shown in Equation (4) and Equation (5):

[0070]

[0071] In formula (4), p t represents the weight of the t-th expert model, represents the weighted sum of the prediction results of each model, N represents the number of expert models participating in the weighting. To ensure the rationality of the probability distribution, it is required that the sum of the weights of all models is 1, so as to ensure that the sum of the final output probabilities is still 1;

[0072] Step 3.5) The fully connected layer of the model classification layer maps the hidden state to the dimension of the number of event categories, using a new weight matrix and bias to perform a linear transformation, as shown in formula (6):

[0073] z cls = W vls + b cls (6)

[0074] The activation function of the classification layer is the sigmoid activation function. This activation function will independently map the value of each output neuron and convert it into a probability value between [0,1]. Each value represents the probability of the corresponding event category, and the probabilities of different event categories are independent of each other, as shown in formula (7), where z represents the input real value, which is calculated by formula (6).

[0075]

[0076] Step 3.6) Use the fine-tuned event detection model of the fusion MoE for event detection of news texts and finally output the event type.

[0077] Furthermore, the specific method of step 4) is as follows:

[0078] Step 4.1) After completing the event type detection in step 3.6), the model constructs an input sequence through a cascaded architecture to achieve structured event argument extraction. The predefined event template library is shown in formula (8):

[0079]

[0080] In formula (8), e i represents the event type, is the corresponding role set, and the parameter m represents the number of argument roles to be extracted under this event. Through the dynamic prompt template based on the event template, the event detection and argument extraction are decoupled into two-level pipeline tasks;

[0081] Step 4.2) For the detected event type e i ∈ Y, the system retrieves its role definition R t, construct a prompt template with clear semantic constraints, and use the constructed prompt template as the input of the event argument extraction module to achieve event argument extraction.

[0082] To verify the effectiveness of the fusion MoE enhanced event detection module, MoE-CNEE was compared with the following five methods on the public datasets ChiFinAnn and DuEE-fin, and the comparison results are shown in Table 1. The specific introduction of the models is as follows:

[0083] (1) DCFEE extracts arguments from a central sentence in the document. It has two variants, DCFEE-O only extracts a single event mention, and DCFEE-M extracts multiple event mentions simultaneously.

[0084] (2) Doc2EDAG uses a path extension strategy to extract multiple events and their corresponding arguments.

[0085] (3) GIT first adopted a graph model that connects sentence nodes and entity nodes in the order of document tokens.

[0086] (4) PTPCG aggregates arguments in the form of a set and extracts events based on these argument sets.

[0087] (5) ReDEE introduced a multi-scale relationship enhanced transformer built on the Transformer framework.

[0088] (6) ProCNet combines proxy nodes, uses a graph model to aggregate sentence and argument information into these nodes, and further optimizes the model using the Hausdorff minimum distance method.

[0089] (7) CAINet models the relationships between various events through an event relationship graph and the correlations between arguments through an argument association graph to effectively aggregate cross-sentence arguments.

[0090] Table 1 Comparative experimental results of event argument extraction of the MoE-CNEE model on public datasets

[0091]

[0092] To verify the effectiveness of the fusion MoE enhanced event detection module, the inventors compared MoE-CNEE with the following five methods on the public datasets ChiFinAnn and DuEE-fin, and the experimental results are shown in Table 2. The specific introduction of the models is as follows:

[0093] (1) BERT, which is an open-source Chinese pre-trained language model developed by Google;

[0094] (2) BERT-WWM. This model is a pre-trained model improved by Harbin Institute of Technology based on Chinese BERT. Its core innovation lies in the adoption of the whole-word masking strategy, which is more in line with the language characteristics of Chinese compared with the single-word masking of the original BERT and can effectively improve the model's ability to understand Chinese semantics.

[0095] (3) Ernie. As a pre-trained language model launched by Baidu, Ernie has significant advantages in semantic understanding by introducing knowledge graphs and multi-task learning mechanisms.

[0096] (4) FinBERT. This model is a Chinese pre-trained language model developed by Entropy Simplification Technology's Artificial Intelligence Laboratory specifically for the financial field. FinBERT can better capture professional terms and specific semantics in the financial field and performs well in financial text processing tasks.

[0097] (5) BERT_BiLSTM_CRF. It uses BERT as a feature extractor and combines BiLSTM and CRF for sequence annotation. This model has shown good performance in event extraction tasks.

[0098] Table 2 Comparison experiment results of MoE-CNEE in event detection on DuEE-fin

[0099]

[0100] The results show that MoE-CNEE is superior to mainstream recommended methods in event detection and argument extraction tasks. Compared with the optimal comparison model, the F1 value of the event detection task of the MoE-CNEE model has increased by 6.28%, and the F1 value of the event argument extraction task has increased by 1.3%. On the ChiFinAnn dataset, the F1 value of this model in the event argument extraction task has increased by 1.2%. These experimental data confirm the effectiveness of the MoE-CNEE model in event extraction tasks.

[0101] Those skilled in the art can easily understand the embodiments of the present invention. The above are only examples of the basic implementation of the present invention and are not intended to limit the present invention.

Claims

1. A cascaded enterprise news event extraction method integrating MoE, characterized in that The method includes the following: Step 1) Obtain corporate news event texts from various financial platforms and news websites, and pre-process the data; Step 2) Adjust the parameters of the large language model, and adjust the parameters of its event detection-related adapter and classification layer; Step 3) The adjusted large model introduces the MoE mechanism to enhance the event detection effect of the model, and performs event detection on the news text by integrating the MoE model; Step 4) Dynamically construct a semantically constrained prompt template based on the event detection results, guide the model to focus on text fragments related to the target role, and complete the extraction of news elements.

2. The cascaded enterprise news event extraction method integrating MoE according to claim 1, characterized in that, The specific method of step 1) is as follows: Step 1.1) Select financial platforms and news websites as data sources and obtain news data regularly; use keyword search or subject search methods to obtain enterprise-related news texts, set the data update frequency, and ensure that the latest enterprise news events are obtained; Step 1.2) Preprocess the data, filter and remove duplicates of advertisements, promotions and non-news content through regular expressions, normalize the text to Unicode, unify the encoding format and remove abnormal characters and invalid punctuation.

3. The cascaded enterprise news event extraction method integrating MoE according to claim 2, characterized in that The specific method of step 2) is as follows: Step 2.1) Freeze the pre-trained model weights, independently train a decomposition matrix, and combine the matrix with the pre-trained model weights to adjust the model. For the original weight matrix W, its change ΔW is approximately expressed as the product of two matrices A and B, as shown in formula (1): ΔW = A * B T (1) Step 2.2) Weight matrix W of the pre-trained model O introduce two matrices A and B, and their adjustment process is shown in formula (2): In Equation (2), A is initialized with random Gaussian values, and B is initialized with 0. At the beginning of training, BA is 0, and a scaling factor is introduced to perform scaling parameters and further optimize the model training process. Here, α represents the scaling amplitude coefficient, which is used to control the overall size of the update amount; r represents the rank of the decomposition matrix.

4. A cascaded enterprise news event extraction method integrating MoE according to claim 3, characterized in that, The specific method of step 3) is as follows: Step 3.1) Treat the event detection task as a multi-label classification task. Given the input text sequence X = [x1, x2,..., x n , the model predicts the activation status of the set of event types Y = {y1, y2,..., y k}, where k represents the total number of event types, and each y i ∈ {0, 1}; Step 3.2) Change the language modeling layer of the model into a classification layer, and add a MoE module in front of the classification layer. This module is jointly composed of multiple expert models and a gating model. The gating model assigns corresponding weights to each expert model. For the input sample, assuming the output of the i-th expert model is o i , and the corresponding true label is t, then the loss function of this model is shown in formula (3): In formula (3), p i represents the weight assigned by the gating model to the i-th expert model. The weight p i is placed outside the loss function, enabling each expert model to independently calculate its loss value and allowing different expert models to compete with each other; Step 3.3) Each expert model classifies and predicts the input data and outputs the probability distribution of each category; the gating model quantifies the importance of each expert model to the final classification result through the assigned weight; Step 3.4) The weights calculated by the gating network are used to weighted aggregate the outputs of each expert model. The prediction result of each expert model is multiplied by its corresponding weight, and these weighted results are summed to obtain the final prediction output. The output form of MoE is shown in formula (4) and formula (5): In Equation (4), p t represents the weight of the t-th expert model, represents the weighted sum of the prediction results of each model. N represents the number of expert models participating in the weighting. The sum of the weights of all models is 1 to ensure that the sum of the final output probabilities is still 1; Step 3.5) The fully connected layer of the model classification layer maps the hidden state to the dimension of the number of event categories, using a new weight matrix and bias for linear transformation, as shown in formula (6): z cls = W cls + b cls (6) Among them, the activation function of the classification layer is the sigmoid activation function. This activation function independently maps the value of each output neuron and converts it into a probability value between [0, 1]. Each value represents the probability of the corresponding event category, and the probabilities of different event categories are independent of each other, as shown in formula (7). Among them, z represents the input real value, which is calculated by formula (6); Step 3.6) Use the adjusted fused MoE event detection model for event detection in news texts, and finally output the event type.

5. The cascade enterprise news event extraction method integrating MoE according to claim 4, characterized in that, The specific method of step 4) is as follows: Step 4.1) After completing the event type detection, the model constructs the input sequence through a cascaded architecture to achieve structured event argument extraction. The predefined event template library is shown in formula (8): In formula (8), e i represents the event type, is the corresponding role set. The parameter m represents the number of argument roles to be extracted under this event. The event detection and event argument extraction are decoupled into two-level pipeline tasks through a dynamic hint template based on the event template; Step 4.2) The detected event type e i ∈ Y, the system retrieves its role definition R from the template library t , constructs a prompt template with explicit semantic constraints, and uses the constructed prompt template as the input of the event argument extraction module to implement the extraction of event arguments.

Images