A military news event extraction method based on an integration method and a confidence score

Abstract

The application discloses a military news event extraction method based on an integrated method and a confidence score, and comprises the following steps: step 1, initializing a model output confidence score distribution; step 2, evaluating the consistency of the model output; step 3, updating the model output confidence score based on the prior and the consistency; and step 4, selecting the optimal model output, in which the model output with the highest model output confidence score is selected as the final output in the iteration process or after the iteration is completed. The application uses a normal distribution to comprehensively represent and estimate the confidence of a single output, estimate the confidence of a single output, and capture the statistical dispersion between the actual confidence and the estimated confidence; important events can be accurately extracted and classified from a large amount of unstructured texts.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology, and in particular relates to a method for extracting military news events based on an integration method and confidence scoring. Background Technology

[0002] In today's increasingly information-driven and networked world, military news, as a unique means of information dissemination, encompasses complex information across multiple levels, including strategy, tactics, and operations. This information holds immense value for national security, military research, and strategic decision-making. Therefore, accurately and efficiently extracting valuable event information from vast amounts of unstructured military news texts has become a crucial issue in the field of information processing.

[0003] Military news texts typically contain rich and complex information, such as location, time, participating entities, and event type. This information is often scattered across different parts of the text and expressed in various ways, posing a significant challenge to event extraction. Furthermore, the language style of military news is usually formal and standardized, potentially containing numerous technical terms and abbreviations, which further complicates text processing.

[0004] Event extraction technology has important applications in multiple fields. For example, Chinese invention application CN116628210A proposes a fault determination method for extracting fault events in smart buildings based on contrastive learning; Chinese invention application CN116757159A proposes an end-to-end multi-task joint document-level event extraction method and system; Chinese invention application CN116631642A provides a method for extracting clinical discovery events; and Chinese invention application CN116579338A provides a document-level event extraction method and system based on ensemble joint learning.

[0005] However, traditional event extraction methods typically rely on predefined rules or templates. While these methods may perform well in specific domains and scenarios, their adaptability and accuracy are often limited when faced with diverse and dynamically changing military news texts. Furthermore, these methods often ignore implicit information and contextual relationships within the text, which to some extent restricts the depth and breadth of event extraction. Ensemble methods aim to improve the accuracy and stability of event extraction by integrating the prediction results of multiple models. However, how to reasonably integrate the outputs of multiple models and how to evaluate the reliability of the outputs of each model remain unsolved problems. Traditional ensemble methods often rely on hard voting or soft voting strategies, which can improve model accuracy to some extent, but still have shortcomings in handling the uncertainty and reliability of model output results.

[0006] In multi-model ensemble scenarios, the confidence score (CS) becomes a crucial metric, reflecting the reliability and certainty of the model's output. However, traditional CS is typically a fixed value, failing to comprehensively reflect the uncertainty and reliability of the model's output. Therefore, accurately evaluating and utilizing CS to improve the accuracy and reliability of event extraction has become an urgent problem to be solved.

[0007] In conclusion, given the complexity and diversity of military news texts, and the various challenges in event extraction, there is an urgent need for a new method capable of accurately, efficiently, and reliably extracting event information. This method should be able to fully mine the potential information within the text, accurately identify and classify event information, and handle the uncertainty and reliability of the model's output results, thereby improving the overall performance of event extraction. Summary of the Invention

[0008] In view of this, the present invention aims to propose a novel ensemble method for extracting events from military news. This method not only estimates the confidence of individual outputs through the expectation of the distribution, but also cleverly captures the statistical dispersion between actual and estimated confidence through standard deviation, thus providing a more comprehensive representation of the confidence score (CS). Furthermore, the present invention discloses a comprehensive CS evaluation method that organically integrates the consistency and prior confidence of multiple model outputs, treating model-level confidence as an initial evaluation of its outputs. This method further enhances the accuracy of CS evaluation through an iterative update algorithm.

[0009] To achieve the above-mentioned objectives, this invention discloses a method for extracting military news events based on an integration method and confidence scoring, comprising the following steps:

[0010] Enter military news;

[0011] Multiple news event extraction models are used to output a set of model outputs about the same fact;

[0012] For each model, an initial confidence level is set for the initial model output, and a confidence score distribution is established based on the confidence level.

[0013] The consistency of each model's output is evaluated according to the consistency comparison rules;

[0014] The model output confidence score is updated based on prior knowledge and consistency.

[0015] During or after the iteration, the model with the highest confidence score is selected, and the military news events extracted by it are used as the final output.

[0016] Furthermore, the initialization model outputs a confidence score distribution, including:

[0017] Using the i-th independent model CP credibility i As a model, the output m i initial expectations Initialize the model output confidence score distribution Where i∈{1,2,…,N} is the index of the independent model, N is the total number of independent models, and μ i and These are the expected value and variance of the credibility, respectively, m i Indicates independent model The output is the extraction result for a specific news event f. Initialized to 0.25.

[0018] Furthermore, the consistency of the evaluation model output includes:

[0019] C i =c e |m e ∈E i , indicating that the model output m i Its external model output set E i The consistent set between them, output set E i Includes excluding m i The outputs of all models that describe the same fact f, except for E, are mathematically expressed as: i ={m e |m e Describe f, e≠i}, c e Determined in the following ways:

[0020]

[0021] in, Indicates the output m of the external model e With m i Under the consistency comparison rule, it is consistent, that is, whether the event types output by the event extraction model are consistent, where Rule represents the consistency comparison rule.

[0022] Furthermore, the update of the confidence score output by the model based on prior knowledge and consistency includes:

[0023] Iteratively update the model output m i The model outputs a confidence score Q(cr) i The following formula is used for iterative updates:

[0024] Q l (cr i )=F({Ql―1 (cr i )∣m i}∪Q l―1 (cr e )∣m e ∈E i C i )

[0025] Where F(·) represents the model output confidence score update function, l represents the l-th iteration, and is a user-defined parameter. F(·) consists of two parts: expectation update and variance update. Q(cr) e ) indicates that the output m is different from the model output. i The confidence distribution of other model outputs, i.e., quality.

[0026] Furthermore, the update expectation μ i The expression is:

[0027]

[0028] Where M represents m i The number of effective external model outputs, i.e., M = |E i |;

[0029]

[0030] The model output m after the (l-1)th iteration e A numerical representation of the confidence distribution quality.

[0031] Furthermore, the updated variance The expression is:

[0032]

[0033] in

[0034]

[0035]

[0036] The model output m after the (l-1)th iteration e The expectation of the confidence distribution. The model output m after the (l-1)th iteration i The expected value of the confidence distribution;

[0037] λ is a user-defined parameter, set to 2. Through the above process, the confidence score of the model output after each iteration is obtained. When the defined number of iterations is reached, the iteration stops, and the final confidence score evaluation result is obtained.

[0038] Furthermore, during or after the iteration, each model outputs m. i The confidence score is determined by Provided. Attached Figure Description

[0039] Figure 1 A framework diagram of the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0041] In the technical solution of this invention, some key symbolic expressions are first defined: A set containing all independent models used for military news event extraction, where each independent model It contains a set of model outputs. Here, i∈{1,2,…,N} is the index of the independent model, and N is the total number of independent models.

[0042] m i : indicates an independent model The output is the extraction result of a specific news event (denoted as fact) f (denoted as model output). In military news event extraction, each model output is m. i It can be viewed as the output of a model, which describes the result or state of a specific event extraction.

[0043] E i For a given model, output m i set E i Includes excluding m i The output of all models that describe the same fact f, except for E. Mathematically, it can be expressed as: E i ={m e |m e Describe f, e ≠ i. Here, M represents m. i The number of effective external model outputs, i.e., M = |E i |

[0044] CP i Independent model The credibility of it is that it is a real number.

[0045] Q(cr i Model output m i The model outputs the probability density function of the confidence score (KQ), and the model follows a normal distribution. Where μ i and These are the expected value and variance of the credibility, respectively.

[0046] Input: Using a set of independent models and the corresponding credibility set CP = {CP1, CP2, ..., CP} N As input, these confidence sets can be given by the model's accuracy on the validation set. The news extraction model is a model in the prior art, such as DMCNN (Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks, ACL 2015), JRNN (Joint Event Extraction via Recurrent Neural Networks, ACL 2016), etc., which will not be described in detail in this invention.

[0047] Target output: The target is to evaluate the output m of each model. i The confidence score. In the scenario of military news event extraction, each model outputs m. i This can be viewed as the output of a model, which describes a specific event extraction result or state. We use these model outputs (i.e., model outputs) and the model's initial confidence to iteratively update the model output confidence score for each model output, and finally select the reliable result (the result with the highest confidence score) as the event extraction result.

[0048] Step a: Initialize the model output confidence score distribution

[0049] Use CP i As model output m i initial expectations Initialize the model output confidence score distribution Where μ i and These are the expected value and variance of the credibility, respectively. Initialized to 0.25.

[0050] Step b: Evaluate the consistency of the model output

[0051] C i =c e |m e ∈E i , indicating that the model output m i Its external model output set E i A consistent set among them. e Determined in the following ways:

[0052]

[0053] in, Indicates the output m of the external model e With m i Consistency is defined under the consistency comparison rule, meaning whether the outputs (event types) of the event extraction models are consistent. The consistency comparison rule means that each model is compared using the same rules.

[0054] Step c: Iteratively update the model output confidence score based on prior knowledge and consistency, updating the model output m. i The model outputs a confidence score Q(cr) i Use the following formula for iterative updates:

[0055] Q l (cr i )=F({Q l―1 (cr i )∣m i}∪Q l―1 (cr e )∣m e ∈E i C i )

[0056] Where F(·) represents the model output confidence score update function,

[0057] l represents the l-th iteration, which is a user-defined parameter. F(·) consists of two parts: expected update and variance update.

[0058] Step c-1: Update the expected μ i The expression is:

[0059]

[0060] in,

[0061]

[0062]

[0063]

[0064] Step c-2: Update variance The expression is:

[0065]

[0066] in

[0067]

[0068]

[0069] The model output m after the (l-1)th iteration e The expectation of the confidence distribution. The model output m after the (l-1)th iteration i The expected value of the confidence distribution;

[0070] λ is a user-defined parameter, typically set to 2. Through the above process, the model output confidence score is obtained after each iteration. The iteration stops after the defined number of iterations is reached, yielding the final confidence score evaluation result.

[0071] Step d: Select the optimal model output

[0072] During or after iteration, the model output with the highest model output confidence score (or credibility) is selected as the final output, where each model output is m. i The confidence score is determined by Provided.

[0073] In this embodiment, it is assumed that there are three military news event extraction models (three independent models), each outputting a set of model outputs regarding the same fact (e.g., whether a military conflict has occurred in a certain region). The outputs of these models are considered independent models. The model output m i Each model's output has an initial confidence level (CP). i This can be determined based on the model's historical performance or expert evaluation. The model is represented as:

[0074]

[0075] m1, m2, and m3 represent the description of fact f by each model. Assume the three outputs are 1, 1, and 2, indicating whether the extracted results conflict (1 for conflict, 2 for no conflict). E i For except m i The set of outputs from other models that describe fact f.

[0076] Step a: Initialize the model output confidence score distribution

[0077] Assign an initial confidence level to the output of each model, for example:

[0078] CP1 = 0.8

[0079] CP2 = 0.6

[0080] CP3 = 0.7

[0081] This invention uses these values ​​as To initialize the model output confidence score distribution Q(cr)i ), Initialized to 0.25.

[0082] Step b: Evaluate the consistency of the model output

[0083] Compare the consistency of each model's output. For example, if two models both predict military conflict in the same region, their outputs are considered consistent. i =c e |m e ∈E i , indicating that the model output m i Its external model output set E i A consistent set among them. e Determined in the following ways:

[0084]

[0085] in, Indicates the output m of the external model e With m i Under the consistency comparison rule, it is consistent, that is, whether the output (event type) of the event extraction model is consistent.

[0086] Step c: Update the model output confidence score based on prior and consistency

[0087] Assume the number of iterations l is 1.

[0088] Step c-1: Update the expected μ i The expression is:

[0089]

[0090] in,

[0091]

[0092]

[0093]

[0094] In a given instance, there are Step c-2: Update variance The expression is:

[0095]

[0096] in

[0097]

[0098]

[0099] In a given instance, we have (σ1) 2 ) 1 =0.26, (σ2) 2 ) 1 =0.26, (σ3) 2 ) 1 =0.28.

[0100] Step d: Select the optimal model output

[0101] During or after iteration, this invention selects the model output with the highest model output confidence score (or credibility) as the final output, where each model output m i The confidence score is determined by Given the example, the confidence scores of the three model outputs are 0.15, 0.00, and -0.08, respectively. Model 1 consistently maintains a high model output confidence score throughout the iteration process, and the occurrence of military conflict is selected as the final military event extraction result.

[0102] This example demonstrates how the proposed technique can be used to integrate outputs from multiple models, improving the accuracy and reliability of military news event extraction by considering the consistency between model outputs and the initial credibility of each model. This approach provides an effective mechanism for evaluating and selecting the most reliable event extraction results when processing outputs from multiple models.

[0103] Compared with the prior art, the present invention has the following beneficial effects:

[0104] This invention proposes a multi-source confidence scoring evaluation algorithm that integrates multiple models to improve the accuracy of military news event extraction, and has the following advantages and positive effects:

[0105] By integrating multiple models, the accuracy and reliability of event extraction have been improved;

[0106] Using a normal distribution to fully represent and estimate the confidence level of a single output not only estimates the confidence level of a single output, but also captures the statistical dispersion between the actual confidence level and the estimated confidence level;

[0107] By integrating the consistency of multi-model outputs and prior confidence, the relevance and reliability of extracted events are ensured.

[0108] The accuracy of confidence assessment was improved by iteratively updating the algorithm;

[0109] In the field of news, it provides a more detailed, accurate, and reliable mechanism for event extraction.

[0110] In summary, the military news event extraction method provided by this invention helps to accurately extract and classify important events from a large amount of unstructured text.

[0111] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0112] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0113] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0114] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for extracting military news events based on an ensemble approach and confidence scoring, characterized in that, Includes the following steps: Enter military news; Multiple news event extraction models are used to output a set of model outputs about the same fact; For each model, an initial confidence level is set for the initial model output, and a confidence score distribution is established based on the confidence level. The consistency of each model's output is evaluated according to the consistency comparison rules; The model output confidence score is updated based on prior knowledge and consistency. During or after the iteration, the model with the highest confidence score is selected, and the military news events extracted by it are used as the final output. The initialization of the model output confidence score distribution includes: using the i-th independent model. Credibility As a model, the output initial expectations Initialize the model output confidence score distribution ,in, It is an index for independent models. It is the total number of independent models. and These are the expected value and variance of the credibility, respectively. Indicates independent model The output is a news event. The extraction results Initialize to 0.25; The consistency of the evaluation model output includes: ,express Its external model output set Consistent set between them, output set Includes except All other descriptions of the same facts The model output is mathematically expressed as: , Determined in the following ways: in, Represents the output of the external model and Under the consistency comparison rule, they are consistent, that is, whether the event types output by the event extraction model are consistent, where Rule represents the consistency comparison rule; The update of the model output confidence score based on prior knowledge and consistency includes: iteratively updating the model output. Model output confidence score Use the following formula for iterative updates: in, This represents the model's output confidence score update function. Indicates the first The next iteration uses custom parameters. Divided into updated expectation and updated variance, This indicates a difference from the model output. The confidence distribution of other model outputs, i.e., quality.

2. The military news event extraction method based on ensemble method and confidence scoring according to claim 1, characterized in that, The update expectation The expression is: in, express The number of effective external model outputs, i.e. It is after the (l-1)th iteration .

3. The method for extracting military news events based on an ensemble method and confidence scoring as described in claim 2, characterized in that, The updated variance The expression is: in The model output after the (l-1)th iteration The expectation of the confidence distribution. The model output after the (l-1)th iteration The expected value of the confidence distribution; The custom parameter is set to 2. Through the above process, the confidence score of the model output after each iteration is obtained. When the defined number of iterations is reached, the iteration stops and the final confidence score evaluation result is obtained.

4. The military news event extraction method based on ensemble method and confidence scoring according to claim 3, characterized in that, During or after the iteration, each model outputs The confidence score is determined by Provided.

Images