A model loss optimization method and system for the long-tail problem of event detection data

Abstract

The present invention discloses a model loss optimization method and system for the long-tail problem of event detection data. The method comprises obtaining a basic model, wherein the basic model is a BERT‑BiLSTM‑CRF model; constructing an optimized loss function of the basic model; using the optimized loss function as a training parameter of the basic model, training the basic model to obtain a trained model; and performing event detection on input data using the trained model. The solution provided by the present invention adopts an optimized loss function for model training, which can significantly improve the performance of the model on long-tail data sets and demonstrate the effectiveness of the model in dealing with data imbalance problems.

Description

Technical Field

[0001] The present invention relates to the field of event detection technology, and in particular to a model loss optimization method and system for the long-tail problem of event detection data. Background Art

[0002] Event detection (ED), according to the definition in the Automatic Content Extraction (ACE) International Evaluation Conference, is a subtask of event extraction, which aims to detect event trigger words and annotate the type of event. For example, in the sentence "John traveled to NYC for a meeting", "traveled" is a trigger word for a "Movement-Transport" event. Event detection is usually the first step in extracting event structured information (event structured information such as arguments such as time, place, and participants; or distinguishing between past, habitual, and future events), and is widely used in multiple downstream fields such as news analysis, social media monitoring, and network security.

[0003] Due to the complexity of the task and the difficulty of building a standard annotation scheme, event detection datasets suffer from the problem of long-tail distribution. The long-tail distribution of data often has a negative impact on the performance of event detection systems because the samples of some event types are far less than other types. This imbalance causes the model to perform poorly in identifying rare events, which are crucial in comprehensive event detection.

[0004] In recent years, researchers have tried to complete event detection tasks by using sequence labeling modeling methods, including Recurrent Neural Network (RNN) and its variants - Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN) and Conditional Random Field (CRF). Pre-trained language models such as BERT have demonstrated state-of-the-art performance on many natural language processing (NLP) tasks. Sequence labeling usually uses BIO (B-begin, I-inside, O-outside) labeling to label each word in a sentence. However, this model faces the problem of data sparsity where non-trigger words are significantly more than trigger words, that is, the text labeled as "O" is significantly more than the text labeled as "B" and "I".

[0005] In summary, data limitations including long-tail distribution problems and data sparsity problems pose huge challenges to event detection tasks. Summary of the invention

[0006] In order to solve the technical problem that the long-tail distribution problem of data and the data sparsity problem bring huge challenges to the event detection task, the embodiment of the present invention provides a model loss optimization method and system for the long-tail problem of event detection data.

[0007] The technical solution of the embodiment of the present invention is achieved as follows:

[0008] An embodiment of the present invention provides a model loss optimization method for the long-tail problem of event detection data, the method comprising:

[0009] Obtain a basic model, where the basic model is a BERT-BiLSTM-CRF model;

[0010] Constructing an optimized loss function of the basic model; using the optimized loss function as a training parameter of the basic model, training the basic model to obtain a trained model;

[0011] Use the trained model to detect events on the input data;

[0012] Among them, the optimization loss function of the basic model is:

[0013]

[0014] in, represents the optimization loss function of the basic model, represents the original loss function of the BERT-BiLSTM-CRF model, It represents the loss function used to alleviate the long-tail distribution of event categories, and its calculation formula is:

[0015]

[0016] in, Represents the total number of event categories, represents the i-th value in the true label sequence, , represents the normalized calculation of the jth value in the response value sequence, represents the normalized calculation of the kth value in the response value sequence, Indicates event category k The weight of

[0017] It represents the loss function used to alleviate the sparseness of categorical labeled data, and its calculation formula is:

[0018]

[0019] in, represents the weight factor, represents the adjustment factor, Represents the jth value in the probability prediction distribution.

[0020] In one embodiment, the original loss function of the BERT-BiLSTM-CRF model is:

[0021]

[0022] in, It represents the expected score of the output sequence obtained by encoding the input sequence through the model. represents the input word sequence, Represents the label sequence corresponding to the input, Represents the probability prediction sequence corresponding to the input sequence.

[0023] In one embodiment, the event category k The weight calculation formula is:

[0024]

[0025] in, represents the adjustment factor, Represents the event category in the dataset k The frequency, is a threshold function, when The output is 1 when the Used to distinguish tail and non-tail categories from the dataset categories; Represents the kth value in the true label sequence.

[0026] In one embodiment, the event category in the data set k The frequency calculation formula is:

[0027]

[0028] in, represents the frequency of category i, Represents the frequency of category i.

[0029] In one embodiment, the BERT-BiLSTM-CRF model includes a BERT model, a BiLSTM model and a CRF model: the BERT model is an embedding layer, which is used to obtain the vector corresponding to the text; the BiLSTM model is a sequence annotation layer, which is used to save long-distance semantic information; the CRF model is a classification layer, which is used to consider the influence of the previous character and the next character on the current character.

[0030] In one embodiment, the BERT model obtains the word embedding representation vector by the following calculation formula:

[0031]

[0032] in, represents the word embedding representation vector, , represents the input word sequence, that is, a single sentence.

[0033] In one embodiment, the BiLSTM model obtains past and future feature information through the following calculation formula:

[0034]

[0035] in, is the forward LSTM representation in the BiLSTM model, It is the backward LSTM representation in the BiLSTM model;

[0036]

[0037]

[0038] in, express The word embedding representation vector.

[0039] An embodiment of the present invention also provides a model loss optimization system for the long-tail problem of event detection data, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor executes the steps of the above-mentioned method when running the computer program.

[0040] This embodiment has the following beneficial effects:

[0041] The solution of this embodiment can significantly improve the performance of the model on long-tail data sets and is effective in dealing with data imbalance problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a flow chart of a model loss optimization method for the long-tail problem of event detection data according to an embodiment of the present invention;

[0043] Figure 2 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0045] The embodiment of the present invention provides a model loss optimization method for the long-tail problem of event detection data. Figure 1 As shown, the method includes:

[0046] Step 101: Obtain a basic model, where the basic model is a BERT-BiLSTM-CRF model;

[0047] Step 102: constructing an optimized loss function of the basic model; using the optimized loss function as a training parameter of the basic model, training the basic model to obtain a trained model;

[0048] Step 103: Use the trained model to perform event detection on the input data.

[0049] Event detection is an information extraction task with practical application value, but the performance of event recognition is poor due to the long-tail distribution problem of the data set. This embodiment regards event trigger word extraction as a sequence labeling problem, and proposes a loss optimization method to solve the problem of long-tail distribution of data in event detection tasks based on sequence labeling. This embodiment adopts the BERT-LSTM-CRF model as the basic architecture, and introduces two loss functions to cope with the challenges brought by data imbalance. First, this embodiment introduces Equalization Loss to balance the data distribution between different event categories and reduce the negative impact of the long-tail effect on model performance. Secondly, in order to deal with the significant imbalance between events and non-events, this embodiment adopts Focal Loss, which further improves the robustness and accuracy of the model by focusing on difficult-to-classify samples. The experimental results show that the combination of these two loss functions can significantly improve the performance of the model on long-tail data sets, demonstrating its effectiveness in dealing with data imbalance problems.

[0050] Specifically, first, this embodiment introduces the basic model used in this embodiment.

[0051] This embodiment regards event detection as a sequence labeling problem based on the BERT-BiLSTM-CRF model. The BERT-BiLSTM-CRF model includes a BERT model, a BiLSTM model (also known as a bidirectional long short-term memory network), and a CRF model. Among them, the BERT model is a pre-trained language model, which is mainly used to obtain the vector corresponding to the text and can well represent the text features. The BiLSTM model is used to save long-distance semantic information. The CRF model is a variant of the hidden Markov model (HMM). The HMM only considers the impact of the previous state on the current state, while the CRF model considers the previous state and the next state. In this embodiment, it is the impact of the previous character on the next character. The impact here can be regarded as a conditional probability, that is, the probability of generating the next character when the previous character is generated.

[0052] This model uses the BERT model as the embedding layer, the Bi-LSTM model as the sequence annotation layer, and then adds a conditional random field (CRF model) as the classification layer for model optimization, fine-tuning the BERT model to make it suitable for this task. In this architecture, the input uses the BIO annotation format to distinguish the beginning (B), internal (I) and (O) words of the trigger word label.

[0053] The BERT model, BiLSTM model and CRF model are introduced below.

[0054] 1.1 BERT Model (Embedding Layer)

[0055] The BERT model is a multi-layer bidirectional Transformer encoder, which is an extension of the original Transformer model. The input representation consists of the concatenation of word embeddings, position embeddings, and paragraph embeddings. A special token "[CLS]" is inserted at the beginning of each sentence, and another special token "[PAD]" is used to standardize the length of the sentence (the [SEP] token is used to separate two sentences and is not involved in this example). The pre-trained BERT model provides powerful contextualized representations and can achieve state-of-the-art performance in many NLP tasks.

[0056] For a sentence , which is input into the BERT model to obtain its word embedding representation vector :

[0057]

[0058] 1.2 BiLSTM model (annotation layer)

[0059] The BiLSTM model is a variant of RNN that effectively helps learn long-range dependencies by using memory cells and forget cells to help control the amount of input to retain or forget from previous states.

[0060] For a given time step Word embedding representation , this embodiment uses BiLSTM to encode features. The forward LSTM is expressed as:

[0061]

[0062] The backward LSTM is represented as:

[0063]

[0064] Then concatenate them into , to capture information from the past and the future.

[0065] 1.3 CRF model (classification layer)

[0066] The feature vector encoded by the BiLSTM model is input into the CRF model to learn the interdependence between the output trigger word labels and find the best label sequence. The CRF model simulates the process of the BiLSTM model using past and future labels to predict the current label. The CRF model defines a transfer matrix , and use the rating To model a pair of consecutive time steps Status to The state transition matrix for the sentence and Tags, in The score at the word is the model parameter The output score For a specific word sequence Corresponding to the label sequence The score is the sum of the transfer score and the network score calculated efficiently through dynamic programming.

[0067]

[0068] Based on the prediction score of CRF, the basic loss of the sequence labeling task is Defined as the difference between the expected score and the predicted score:

[0069]

[0070] Through the introduction of the basic model, we can know that is the original loss function of the BERT-BiLSTM-CRF model. The long-tail problem of data in event detection tasks affects the performance of event detection models. In event detection methods based on sequence labeling patterns, the limitations of data are mainly divided into the imbalance between classes of each event itself and the huge difference between event labels and non-event labels in data labels. In order to alleviate the challenges brought by long-tail distribution and data sparsity, this embodiment proposes a novel loss optimization method. From the perspective of loss optimization, this embodiment designs Equalization Loss to alleviate the long-tail distribution problem presented by the event category itself, and also designs Focal Loss to alleviate the problem of text with category annotations and its sparsity caused by the sequence labeling paradigm. The essence of these two losses is to balance the few-sample category and the many-sample category by adjusting the loss weights between the categories.

[0071] Specifically, about the Equalization Loss of alleviating the long-tail distribution of event categories:

[0072] In event detection tasks, especially when dealing with data sets with long-tail distributions, the problem of class imbalance is often encountered. In order to meet this challenge, this embodiment adopts a novel loss function, EqualizationLoss. The core point is to ignore the side effect gradient of negative samples for rare categories whose frequency is lower than the threshold, thereby ensuring the training of the model for a small number of samples. Therefore, this embodiment adopts Equalization Loss to improve the performance of the model on data sets with an unbalanced number of event categories by adjusting the loss of rare event categories.

[0073] First, this embodiment obtains the Logits sequence output by the model according to the encoding vector of the model, and then calculates the probability prediction distribution of the model:

[0074]

[0075]

[0076] Among them, FC is the full connection classification layer. is the model's predicted Logits sequence, which is converted into a probability distribution through the Softmax function .

[0077] Next, this embodiment first counts the number of samples in each category in the statistical data set. , which can use these counts to calculate the weight of each category and deal with the problem of class imbalance.

[0078] Then, the frequency values ​​of each class of samples are calculated as follows:

[0079]

[0080] in, It is the frequency value calculated based on the total number of samples and the number of samples in each category.

[0081] For event categories The weights are calculated as follows:

[0082]

[0083] in, is the regulating factor, Indicates the categories in the dataset The frequency, is a threshold function, when The output is 1 when it is, otherwise it is 0. Used to distinguish tail categories (few sample categories) and non-tail categories from the dataset categories.

[0084] Finally, the Equalization Loss is calculated based on the weights of each event category as follows:

[0085]

[0086] This loss function is essentially a variant of the cross entropy loss function based on the Sigmoid function by introducing the weight parameter ω.

[0087] In addition, about alleviating the sparsity of categorical labeled data with Focal Loss:

[0088] Focal Loss aims to give greater weight to samples that are difficult to classify, so that the model pays more attention to samples that are difficult to predict during training. More specifically, Focal Loss is a loss function proposed to solve the problem of data imbalance between foreground categories (positive samples) and background categories (negative samples) in target detection tasks. Through analysis, it is found that Focal Loss can be well used to deal with the problem of sparse annotations in event detection tasks. In sequence annotation tasks, text labeled O can be analogous to background categories (i.e., targets that do not need to be paid attention to), and text labeled B and I can be used as foreground categories (i.e., target categories that need to be detected). Therefore, by treating data labeled O and non-O as negative samples and positive samples of the loss function respectively, the Focal Loss function can well alleviate the problem of sparse annotation data.

[0089] Similarly, this embodiment first outputs the Compute the probability prediction distribution of the model output:

[0090]

[0091]

[0092] Among them, FC is the full connection classification layer. is the model's predicted Logits sequence, which is converted into a probability distribution through the Softmax function .

[0093] Then, in order to solve the imbalance between positive and negative samples, a weight factor is introduced into the loss function :

[0094]

[0095] However, since easy-to-classify negative samples occupy the main loss value and dominate the loss gradient during training, the weight factor Simply balancing the weights of positive and negative samples is not enough for model optimization. Therefore, a further adjustment factor is introduced on this basis. The purpose is to reduce the loss ratio of easy-to-classify samples and focus on the loss ratio of difficult-to-classify positive samples. The calculation formula of Focal Loss with the adjustment factor is as follows:

[0096]

[0097] Combining these two balancing methods, the final loss function is:

[0098]

[0099] This loss function is essentially a variant of the cross entropy loss function for binary classification tasks that introduces a conditioning factor.

[0100] Through the introduction of the Equalization Loss for alleviating the long-tail distribution of event categories and the Focal Loss for alleviating the sparsity of categorical labeled data, this embodiment can combine the three loss functions to obtain the total loss function of the model (i.e., the optimized loss function):

[0101]

[0102] This embodiment regards event trigger word extraction as a sequence labeling problem, adopts the BERT-BiLSTM-CRF model as the basic model (this is a powerful architecture that performs well in sequence labeling tasks), and introduces two specially designed loss functions. First, in order to solve the long-tail problem of the data set itself, this embodiment introduces EqualizationLoss to balance the data distribution between different event categories, aiming to reduce the negative impact of the long-tail effect on model performance. In addition, in order to solve the sparse annotation problem caused by the sequence labeling paradigm, this embodiment uses Focal Loss, focusing on classified sample data, balancing the proportion of unclassified samples, and further improving the robustness and accuracy of the model.

[0103] In addition, in order to verify the effect of the solution of this embodiment, an evaluation was performed on the ACE2005 event extraction benchmark dataset.

[0104] 1.1 Dataset

[0105] The dataset annotates 8 event types and 33 sub-events, and is formalized as a 67-category annotation task under BIO annotation. In order to make a reasonable comparison, this example uses the same data partitioning as the official benchmark. Table 1 contains the statistical information of the training, validation, and test data partitioning of the dataset.

[0106] Table 1 ACE2005 dataset division

[0107]

[0108] 1.2 Evaluation indicators

[0109] This embodiment uses three evaluation indicators of event classification: precision, recall, and F1 value, and takes the average value of five experiments of the model on the data set as the final result.

[0110] 1.3 Parameter settings

[0111] The hyperparameters for the best performance of the model are set as follows, which are selected by random search and performance on the validation dataset. The BERT-base model in this example is used as the embedding layer, with an input length of 128 and a hidden state dimension of 768. For BiLSTM, this example uses a single-layer network with 200 hidden units for encoding. This example uses a dropout rate of 0.5, uses the Adam optimizer for optimization, sets the learning rate to 0.0001, epochs to 100, and batchsize to 20. Set the frequency threshold to 100 and the balance factor to 0.5 in the Equalization Loss. Use a balance factor of 0.25 and a scaling factor of 2.0 in the Focal Loss.

[0112] 2. Comparison Models

[0113] The method of this embodiment is compared with other current methods for event detection based on pre-training models, including:

[0114] 1) BERT model: A multi-classification method based on BERT model fine-tuning.

[0115] 2) BERT-CRF model: Based on the BERT-base model, a CRF classification layer is added.

[0116] 3) RoBERTa model: A fine-tuning method based on the RoBERTa-base pre-trained language model.

[0117] 3. Event Detection Experiment Results

[0118] This embodiment evaluates the method proposed in this embodiment on the ACE2005 dataset, and the results are shown in Table 2, where BERT-BiLSTM-CRF+LO represents the method of adding loss optimization to the basic model in this paper. The experimental results show that the BERT-BiLSTM-CRF+LO model performs best in the sequence labeling task. Compared with the fine-tuning and BERT-CRF models using only BERT-base, the method of this embodiment has significant improvements in all evaluation indicators. Specifically, the P value of the BERT-BiLSTM-CRF+LO model is 72.6%, the R value is 67.5%, and the F1 value is 70.0%, all of which are higher than other methods. This shows that the BERT model combining BiLSTM and CRF can better capture the contextual information in the sequence and achieve a better balance between labeling accuracy and recall. Therefore, adding the loss optimization method of this article to the model when processing sequence labeling tasks provides a more superior implementation solution.

[0119] Table 2 Experimental results on ACE2005 dataset

[0120]

[0121] ∗ indicates that the above results are all reproduced based on the pre-trained model

[0122] 4. Ablation Experiment Results

[0123] Ablation analysis is performed on the loss optimization method proposed in this embodiment to evaluate the contribution of each component to the model performance. Table 3 shows the results of the ablation experiment (where w / o means removing the component). The results show that after removing EQL, the P value, R value and F1 value of the model dropped to 72.0%, 63.1% and 67.2% respectively, indicating that Equalization Loss plays an important role in improving performance. When Focal Loss is removed, the F1 value drops to 68.3, a decrease of 1.7, indicating that the loss optimization method also significantly affects the model performance. When these two components are removed at the same time, that is, only the basic BERT-BiLSTM-CRF model is used, the model performance drops significantly to an R value of 59.4 and an F1 value of 67.0, while the P value rises to 76.7%, indicating that the use of sequence labeling method sacrifices the recall rate to achieve better overall performance. These results show that Equalization Loss and Focal Loss complement each other and jointly improve the overall performance of the model, demonstrating their effectiveness in alleviating the long-tail problem of data. In addition, these two types of loss optimization methods can well balance the recall rate and precision rate, highlighting their key role in the model.

[0124] Table 3 Ablation experiment results

[0125]

[0126] This embodiment is tested on the ACE2005 dataset with a long-tail problem. The results show that the method of this embodiment significantly improves the performance of the model in processing long-tail datasets. This loss optimization strategy provides a new perspective for solving the problem of data imbalance and has broad application potential.

[0127] This embodiment proposes a loss optimization method based on the BERT-BiLSTM-CRF model to address the long-tail problem of data in event detection tasks. By introducing Equalization Loss and balancing the data distribution of different event categories, the negative impact of the long-tail effect on model performance is reduced. At the same time, Focal Loss is used to focus on samples that are difficult to classify, alleviating the limitations caused by data sparsity, thereby improving the robustness and accuracy of the model. Experimental results show that combining these two loss functions can significantly improve the performance of the model on long-tail data sets, effectively solve the problem of data imbalance, and improve the accuracy of event detection, indicating that the method of this embodiment has good performance when processing long-tail data.

[0128] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a model loss optimization system for the long-tail problem of event detection data, including: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the above-mentioned method are executed.

[0129] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0130] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0131] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 2As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0132] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0140] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0143] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model loss optimization method for the long-tail problem of event detection data, characterized in that: The method comprises: Obtain a basic model, where the basic model is a BERT-BiLSTM-CRF model; Constructing an optimized loss function of the basic model; using the optimized loss function as a training parameter of the basic model, training the basic model to obtain a trained model; Use the trained model to detect events on the input data; Among them, the optimization loss function of the basic model is: in, represents the optimization loss function of the basic model, represents the original loss function of the BERT-BiLSTM-CRF model, It represents the loss function used to alleviate the long-tail distribution of event categories, and its calculation formula is: in, Represents the total number of event categories, represents the i-th value in the true label sequence, , represents the normalized calculation of the jth value in the response value sequence, represents the normalized calculation of the kth value in the response value sequence, Indicates event category k The weight of It represents the loss function used to alleviate the sparseness of categorical labeled data, and its calculation formula is: in, represents the weight factor, represents the adjustment factor, Represents the jth value in the probability forecast distribution.

2. The model loss optimization method for the long-tail problem of event detection data according to claim 1 is characterized in that: The original loss function of the BERT-BiLSTM-CRF model is: in, It represents the expected score of the output sequence obtained by encoding the input sequence through the model. represents the input word sequence, Represents the label sequence corresponding to the input, Represents the probability prediction sequence corresponding to the input sequence.

3. The model loss optimization method for the long-tail problem of event detection data according to claim 1 is characterized in that: Event Category k The weight calculation formula is: in, represents the adjustment factor, Represents the event category in the dataset k The frequency, is a threshold function, when The output is 1 when the Used to distinguish tail and non-tail categories from the dataset categories; Represents the kth value in the true label sequence.

4. The model loss optimization method for the long-tail problem of event detection data according to claim 3 is characterized in that: Event categories in the dataset k The frequency calculation formula is: in, represents the frequency of category i, represents the frequency of category i.

5. The model loss optimization method for the long-tail problem of event detection data according to claim 3 is characterized in that: The BERT-BiLSTM-CRF model includes a BERT model, a BiLSTM model and a CRF model: the BERT model is an embedding layer, which is used to obtain the vector corresponding to the text; the BiLSTM model is a sequence annotation layer, which is used to store long-distance semantic information; and the CRF model is a classification layer, which is used to consider the influence of the previous character and the next character on the current character.

6. The model loss optimization method for the long-tail problem of event detection data according to claim 5 is characterized in that: The BERT model obtains the word embedding representation vector through the following calculation formula: in, represents the word embedding representation vector, , represents the input word sequence, that is, a single sentence.

7. The model loss optimization method for the long-tail problem of event detection data according to claim 5 is characterized in that: The BiLSTM model obtains past and future feature information through the following calculation formula: in, is the forward LSTM representation in the BiLSTM model, It is the backward LSTM representation in the BiLSTM model; in, express The word embedding representation vector.

8. A model loss optimization system for the long-tail problem of event detection data, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 7 are executed.

Images