Method and apparatus for event extraction in the financial field based on lifelong learning

Abstract

The present invention discloses a method and device for event extraction in the financial field based on lifelong learning. The method includes: configuring an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type; training the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module. Through the present invention, the technical problem of low efficiency of the model in extracting event information in the related art is solved, and the adaptability and generality of the event extraction model are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a method and device for event extraction in the financial field based on lifelong learning. Background Art

[0002] In the related art, with the rapid development of information technology, how to quickly extract key and effective information from a large number of news events has become the primary problem faced by researchers. Guided by such a need, information extraction came into being. Information Extraction refers to automatically extracting structured information from unstructured sources, and such information can be entities, entity relationships, entity attributes, etc. Event Extraction is a more complex form of information extraction, which can provide a higher-level content processing abstraction ability. It is an important research direction in natural language processing and a subtask of information extraction technology, aiming to extract trigger words and elements that can describe events from text, and plays a very important role in the field of knowledge mining.

[0003] Related concepts include: Event: Something that occurs within a specific time segment and geographical range, involving one or more roles and consisting of one or more actions, generally at the sentence level. Event trigger word: The core word indicating the occurrence of an event, mostly verbs or nouns; Event type: Pre-defined event types, such as investment, judgment, acquisition, etc.; Event elements: The participants in an event, mainly composed of entities, values, and times. A value is a non-entity event participant; Element role: The role played by an event element in an event, such as investor, investee, etc.

[0004] For example: XX Technology once received XX investment from XX Capital in October 2017. As shown in Table 1:

[0005] Table 1

[0006]

[0007] Event extraction is an important research direction in natural language processing and a subtask of information extraction technology, aiming to extract trigger words and elements that can describe events from text, and plays a very important role in the field of knowledge mining. Especially with the development of deep learning, more and more neural network models show better effectiveness and accuracy in text processing. The task of identifying the occurrence of events from open-domain free text and extracting various elements of the events has become a research difficulty in text information extraction and mining.

[0008] Event extraction includes domain-specific event extraction and open-domain event extraction. Domain-specific event extraction refers to pre-defining the types of target events and the specific structures of each type (including specific event elements) before extraction, and usually a certain amount of labeled data is given. Open-domain event extraction means that the possible event types and event structures are unknown before event recognition. Therefore, this task usually has no labeled data and mainly relies on unsupervised methods and distributional hypothesis theory.

[0009] Financial event extraction is the application of event extraction technology in the financial field, which can help investors quickly obtain the main events of a company, identify investment risks and opportunities, have a more comprehensive understanding of the financial market, and then make correct investment decisions, etc. However, due to the exponential growth of the number of financial texts, the high timeliness of relevant texts, the complexity of industry terms, and the fact that the texts often contain a large amount of noisy texts and irrelevant financial entities, the difficulty of event extraction in the financial field has increased significantly. Traditional financial event extraction methods need to extract different elements for different event types, and information cannot be shared between categories, increasing the difficulty and complexity of event extraction.

[0010] When extracting time in related technologies, the following defects also exist: (1) Most current studies on domain-specific event extraction are for specific data sets, and different event elements need to be extracted for different event types in this domain. Information cannot be shared between different event categories, increasing the difficulty and complexity of event extraction. (2) Most current financial event extraction models are trained separately for specific learning tasks, and the models trained in specific tasks cannot perform well on new tasks, and their memory function and transfer ability for knowledge are not ideal. (3) There is a lack of a feedback mechanism for event detection and event element extraction results, making it difficult to correct misclassifications and element extraction results in a timely manner, and the event extraction template cannot be updated in a timely manner, making it impossible to achieve the generalization and scalability of the model.

[0011] No effective solution has been found for the above problems existing in the related technologies. Summary of the Invention

[0012] An embodiment of the present invention provides a method and device for financial event extraction based on lifelong learning.

[0013] According to one aspect of the embodiments of the present application, a method for financial event extraction based on lifelong learning is provided, including: configuring an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type; training the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module.

[0014] Furthermore, training the initial event extraction template using the sample text information includes: for each description text in the sample text information, inputting the description text into the event detection module, outputting the event type information of the description text, and training the event detection module; inputting the event type information and the description text into the element extraction module, outputting the element role information of the description text, and training the element extraction module.

[0015] Furthermore, the event detection module includes a first word embedding layer, a first Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a first Bidirectional Long Short-Term Memory (BiLSTM) layer, and a first Conditional Random Field (CRF) layer. Inputting the description text into the event detection module and outputting the event type information of the description text includes: inputting the description text into the first word embedding layer to obtain the text vector of the description text; using the first BERT encoding layer to perform embedding encoding on the text vector to obtain the word vector matrix of the description text; inputting the word vector matrix into the first BiLSTM layer to output the first probability matrix of the description text, where the first probability matrix includes the probability that each word in the word vector matrix is mapped to each event type label; using the first CRF layer to obtain the optimal label from the probability matrix and determining the optimal label as the event type information of the description text.

[0016] Furthermore, the element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. Inputting the event type information and the description text into the element extraction module and outputting the element role information of the description text includes: inputting the description text into the second word embedding layer to obtain the element vector matrix of the description text; for each element vector in the element vector matrix, using the second BERT encoding layer to perform embedding encoding on the element vector to obtain the role vector matrix of the event element, where the role vector matrix includes the element text positions where the event element belongs to each element role; inputting the role vector matrix into the second BiLSTM layer to output the second probability matrix of the event element, where the second probability matrix includes the probability that each event element in the role vector matrix is mapped to each element role label; using the second CRF layer to obtain the optimal label from the second probability matrix and determining the optimal label as the element role information of the event element.

[0017] Further, the second BERT encoding layer includes multiple groups of binary classifiers. Using the second BERT encoding layer to perform embedding encoding on the element vectors to obtain a role vector matrix of event elements, which includes: inputting each vector identifier token of the element vectors into each group of binary classifiers respectively, and outputting the probability values of the start character and end character of each identifier of the element vectors belonging to the corresponding element role, where each binary classifier corresponds to an element role.

[0018] Further, the initial event extraction template includes an event detection module and an element extraction module. Training the initial event extraction template using the sample text information includes: encoding the input sample text information using the first BERT encoding layer of the event detection module to output event type information, and training the event detection module; reading the module parameters of the trained event detection module and the word vectors of each sentence in each description text in the sample text information; concatenating the word vectors with the vectors corresponding to the event type and inputting them into the second BERT encoding layer of the element extraction module, fixing the and randomly initializing it as the module parameters of the element extraction module ; where the event detection module includes a first word embedding layer, the first BERT encoding layer, a first BiLSTM layer, and a first CRF layer, the element extraction module includes a second word embedding layer, the second BERT encoding layer, a second BiLSTM layer, and a second CRF layer, the second BiLSTM layer receives inputs from the second BERT layer and the first BERT layer of the event detection module simultaneously through horizontal connection, and the first BERT layer and the second BERT layer are interconnected.

[0019] Further, after outputting the element role information of the description text, the method further includes: in response to a correction instruction, replacing the event type information and the element role information with correct event type information and correct element role information; inputting the replaced correct event type information, the correct element role information, and the description text into the element extraction module again, outputting the element role information of the description text, and continuing to train the element extraction module.

[0020] According to another aspect of the embodiments of the present application, there is also provided a financial domain event extraction device based on lifelong learning, including: a configuration module for configuring an initial event extraction template, where the initial event extraction template includes multiple event types, and element roles corresponding to each event type; a construction module for training the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module.

[0021] Further, the building block includes: a first training unit configured to input each description text in the sample text information into the event detection module, output event type information of the description text, and train the event detection module; and a second training unit configured to input the event type information and the description text into the element extraction module, output element role information of the description text, and train the element extraction module.

[0022] Further, the event detection module includes a first word embedding layer, a first Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a first Bidirectional Long Short-Term Memory (BiLSTM) layer, and a first Conditional Random Field (CRF) layer. The first training unit includes: an input subunit configured to input the description text into the first word embedding layer to obtain a text vector of the description text; an encoding subunit configured to perform embedding encoding on the text vector by using the first BERT encoding layer to obtain a word vector matrix of the description text; a processing subunit configured to input the word vector matrix into the first BiLSTM layer to output a first probability matrix of the description text, where the first probability matrix includes probabilities of each word in the word vector matrix being mapped to each event type label; and an obtaining subunit configured to obtain an optimal tag from the probability matrix by using the first CRF layer and determine the optimal tag as the event type information of the description text.

[0023] Further, the element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. The second training unit includes: an input subunit configured to input the description text into the second word embedding layer to obtain an element vector matrix of the description text; an encoding subunit configured to perform embedding encoding on each element vector in the element vector matrix by using the second BERT encoding layer to obtain a role vector matrix of event elements, where the role vector matrix includes element text positions where the event elements belong to each element role; a processing subunit configured to input the role vector matrix into the second BiLSTM layer to output a second probability matrix of the event elements, where the second probability matrix includes probabilities of each event element in the role vector matrix being mapped to each element role label; and an obtaining subunit configured to obtain an optimal tag from the second probability matrix by using the second CRF layer and determine the optimal tag as the element role information of the event elements.

[0024] Further, the second BERT encoding layer includes multiple groups of binary classifiers, and the encoding subunit is further configured to: input each vector identification token of the feature vector into each group of binary classifiers respectively, and output the probability values that each identification of the feature vector belongs to the start character and the end character of the corresponding feature role of the feature, where each binary classifier corresponds to a feature role.

[0025] Further, the construction module includes: an encoding unit, configured to encode the input sample text information by using the first BERT encoding layer of the event detection module, and output event type information to train the event detection module; a reading unit, configured to read the module parameters of the trained event detection module and the word vectors of each sentence in each description text in the sample text information; a processing unit, configured to splice the word vectors with the vectors corresponding to the event type and then input the spliced vectors into the second BERT encoding layer of the feature extraction module, and fix the and randomly initialize it as the module parameters of the feature extraction module ; where the event detection module includes a first word embedding layer, the first BERT encoding layer, a first BiLSTM layer, and a first CRF layer, the feature extraction module includes a second word embedding layer, the second BERT encoding layer, a second BiLSTM layer, and a second CRF layer, the second BiLSTM layer receives inputs from both the second BERT layer and the first BERT layer of the event detection module through horizontal connection, and the first BERT layer and the second BERT layer are connected to each other.

[0026] Further, the construction module further includes: an error correction unit, configured to, after the second training unit outputs the feature role information of the description text, in response to an error correction instruction, replace the event type information and the feature role information with correct event type information and correct feature role information; an iteration unit, configured to input the replaced correct event type information, the correct feature role information, and the description text into the feature extraction module again, output the feature role information of the description text, and continue to train the feature extraction module.

[0027] According to another aspect of the embodiments of the present application, there is also provided a storage medium, which includes a stored program, and the program executes the above steps when running.

[0028] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory communicate with each other through the communication bus; where: the memory is used for storing a computer program; the processor is used for executing the steps in the above method by running the program stored on the memory.

[0029] The embodiment of the present application also provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the steps in the above method.

[0030] Through the present invention, an event extraction model based on an event template is realized. First, it can be well generalized to new event categories. Second, it can share the underlying information of event classification and element extraction. At the same time, it avoids the transmission of incorrect information between tasks. Finally, it enables the model to simultaneously focus on the correspondence between multiple problems and texts, realizes the parallel extraction of event elements, solves the technical problem of low efficiency of event information extraction by the model in the related art, and improves the adaptability and generality of the event extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention, form a part of the present application, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation to the present invention. In the drawings:

[0032] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;

[0033] Figure 2 is a flowchart of a method for extracting events in the financial field based on lifelong learning according to an embodiment of the present invention;

[0034] Figure 3 is a structure diagram of event extraction based on lifelong learning according to an embodiment of the present invention;

[0035] Figure 4 is a flowchart of event extraction based on lifelong learning according to an embodiment of the present invention;

[0036] Figure 5 is a structure block diagram of a device for extracting events in the financial field based on lifelong learning according to an embodiment of the present invention;

[0037] Figure 6 is a structure block diagram of an electronic device for implementing an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application. It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] Embodiment 1

[0041] The method embodiment provided by the first embodiment of this application can be executed on a server, a computer, a mobile phone, or a similar computing device. Taking running on a server as an example, Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention. As Figure 1 shown, the server may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above-mentioned server may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that, Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned server. For example, the server may further include more or fewer components than those Figure 1 shown in the figure, or have a different configuration from that Figure 1 shown in the figure.

[0042] The memory 104 can be used to store server programs, such as software programs and modules of application software, such as the server program corresponding to a financial domain event extraction method based on lifelong learning in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely provided with respect to the processor 102, and these remote memories may be connected to the server through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the server. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0044] In this embodiment, a financial domain event extraction method based on lifelong learning is provided. Figure 2 It is a flowchart of a financial domain event extraction method based on lifelong learning according to an embodiment of the present invention, as Figure 2 shown, and the process includes the following steps:

[0045] Step S202, configure an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type;

[0046] Taking the event extraction in the financial domain as an example, the present invention considers the generalization of event elements in the financial domain. Among different event types, some elements are common and can directly share the same event element roles, such as time, location, etc. And some elements have different element role names in different event types but similar natures, such as listed companies, executing companies, etc., which all belong to the relevant organizations that initiate events. Therefore, in view of these characteristics, the present invention constructs a set of event extraction templates for the financial domain, and weakens the element roles with category differences by mining the coupling between roles, so that the data with the same nature becomes shared data, which can be regarded as the data expansion of each role, and at the same time enhances the recognition performance of relevant roles.

[0047] The present invention designs a general event template for the financial field based on multiple datasets extracted from public financial field events, which is mainly divided according to the grammatical relationship between event elements and trigger words, covering all event types. The datasets include: "CCKS XX: Small-Sample Cross-Class Transfer Event Extraction for the Financial Field", "CCKS XX: Document-Level Event Subject and Element Extraction for the Financial Field", "XX Language and Intelligent Technology Competition: Event Extraction Task", and the financial field dataset open-sourced by the Doc2EDAG model on github. In the original event template, the number of elements corresponding to different event types is different, while the element roles in the event extraction template constructed by us can cover the event elements under all event types. The template designed by the present invention has a total of 17 event categories and 13 element roles, as shown in Table 2:

[0048] Table 2

[0049]

[0050] In step S204, the initial event extraction template is trained using the sample text information, and the target event extraction template is output. Among them, the initial event extraction template includes an event detection module and an element extraction module.

[0051] The event extraction based on the template in this embodiment includes two subtasks: event detection and element extraction, corresponding to the event detection module and the element extraction module respectively. Among them, event detection can be divided into two subtasks: trigger word recognition and event type classification. The element extraction task includes event element recognition and element role classification. Event element recognition is to judge the event type to which each word in the sentence belongs, and the element role recognition task is to judge the role of the given element in the sentence based on a specific event type.

[0052] Through the above steps, an event extraction model based on an event template is implemented. First, it can be well generalized to new event categories. Second, it can share the underlying information of event classification and element extraction, while avoiding the transmission of incorrect information between tasks. Finally, it can enable the model to simultaneously focus on the correspondence between multiple problems and texts, realize the parallel extraction of event elements, solve the technical problem of low efficiency of event information extraction in the related technology, and improve the adaptability and generality of the event extraction model.

[0053] In an application scenario, when extracting events from financial domain event texts, the overall model is divided into three parts: event detection, element extraction, and lifelong learning. Both event detection and element extraction rely on the feature representations learned by BERT, and a BiLSTM is used to learn the sentence semantic information and sequence position information, where BiLSTM represents a bidirectional long short-term memory model, which is a special type of recurrent neural network. Lifelong learning mainly inputs the relevant parameters output from the BERT layer of event detection into the BiLSTM layer of element extraction to achieve the lifelong learning of the model.

[0054] In this embodiment, training the initial event extraction template using sample text information includes:

[0055] S11. For each description text in the sample text information, input the description text into the event detection module, output the event type information of the description text, and train the event detection module;

[0056] In one implementation, the event detection module includes a first word embedding layer, a first Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a first Bi-directional Long Short-Term Memory (BiLSTM) layer, and a first Conditional Random Field (CRF) layer. Inputting the description text into the event detection module and outputting the event type information of the description text includes: inputting the description text into the first word embedding layer to obtain the text vector of the description text; using the first BERT encoding layer to perform embedding encoding on the text vector to obtain the word vector matrix of the description text; inputting the word vector matrix into the first BiLSTM layer to output the first probability matrix of the description text, where the first probability matrix includes the probability that each word in the word vector matrix maps to each event type label; using the first CRF layer to obtain the optimal label from the probability matrix and determining the optimal label as the event type information of the description text.

[0057] Taking the text "IBXX acquires XX Company" as an example, the input of the event detection module is the text information, and the output is the category to which the event in the text belongs. In the above example, the event category - "acquisition" will be output. This event detection module includes: a word embedding layer, a BERT encoding layer, a Bi-directional Long Short-Term Memory (BiLSTM) layer, and a Conditional Random Field (CRF) layer.

[0058] The word embedding layer is used to obtain the vector representation of each word in the text information in a high-dimensional space, and the word vector is:

[0059]

[0060] Among them represents the dimension of the word vector. Then the text can be represented as:

[0061]

[0062] Among them represents the matrix composed of word vectors, represents the length of the sentence. Then, here a pre-trained BERT model is used to perform embedding encoding on the text, and the hidden vector of the last layer is obtained as the representation vector of each word.

[0063] The word vector matrix output by the BERT encoding layer can obtain more semantic information through the BiLSTM model. LSTM is a long short-term memory neural network, and the input is a vector matrix. The vector representation of the hidden layer can be obtained through the following steps:

[0064] (3)

[0065] (4)

[0066] (5)

[0067] (6)

[0068] Among them, represents the Sigmoid function, respectively represent the input gate, forget gate, output gate, and the final Cell. The bidirectional long short-term memory neural network (BiLSTM) passes a sequence through the forward LSTM and the backward LSTM respectively, so as to obtain two different hidden layer representations in the forward and backward directions and , and combine the two to obtain the output of the BiLSTM, which is the representation information related to the context of each word. This representation information contains the representation features related to the specific downstream tasks.

[0069] The probability matrix transmitted by the BiLSTM is used to obtain the optimal label of the event through the method of conditional random field CRF. Let the input BiLSTM output probability matrix be Among them, represents the probability that the th character is mapped to the th label. When the predicted label sequence of the known sequence is , then the score of the current sequence is:

[0070]

[0071] Among them, is the transition probability matrix, indicating the label transfers to When, by solving the maximum value of can obtain the optimal label sequence, so as to detect the category to which the event belongs.

[0072] S12, input the event type information and the description text into the element extraction module, output the element role information of the description text, and train the element extraction module.

[0073] Still taking the text "IBXX acquires XX Company" as an example, the input of the element extraction module is the classification information output by the event detection module and the text information evaluated by experts, and the output is the roles corresponding to each element in the text. In the above example, the input of this module is: event type "acquisition", word segmentation "IBXX", "acquires" and "XX Company", and the role types of the output elements are: Sub-org("IBXX"), trigger("acquires"), Obj-org("XX Company").

[0074] Given the event type, event element extraction is to extract the elements related to the event type and the roles played by these elements. Due to the fact that most of the elements of the event are long noun phrases and the problem of role overlap, this invention adds multiple sets of binary classifiers on BERT, and each set of classifiers serves a role to determine the scope of all elements belonging to it. This module is also divided into four parts: word embedding layer, BERT encoding layer, bidirectional long short-term memory model (BiLSTM) layer, and conditional random field (CRF) layer.

[0075] In one of the implementation manners, the element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. Input the event type information and the description text into the element extraction module, and output the element role information of the description text, including: input the description text into the second word embedding layer to obtain the element vector matrix of the description text; for each element vector in the element vector matrix, use the second BERT encoding layer to perform embedding encoding on the element vector to obtain the role vector matrix of the event element, where the role vector matrix includes the element text positions where the event element belongs to each element role; input the role vector matrix into the second BiLSTM layer to output the second probability matrix of the event element, where the second probability matrix includes the probabilities that each event element in the role vector matrix maps to each element role label; use the second CRF layer to obtain the optimal label from the second probability matrix, and determine the optimal label as the element role information of the event element.

[0076] In an optional example, the second BERT encoding layer includes multiple groups of binary classifiers. The second BERT encoding layer is used to perform embedding encoding on the feature vectors to obtain the role vector matrix of event features, including: inputting each vector identifier token of the feature vectors into each group of binary classifiers respectively, and outputting the probability values of each identifier of the feature vectors belonging to the start character and end character of the corresponding feature role, where each binary classifier corresponds to a feature role.

[0077] Since features and roles are separated, a feature can play multiple roles, and a token can also belong to different features. Then the token is predicted to be the role of the start probability of the feature is:

[0078]

[0079] is predicted to be the end probability of:

[0080]

[0081] where the subscript represents start, the subscript represents end, is the weight of the binary classifier, the purpose is to detect the start of the feature playing the role of the feature, is to detect the end of the feature playing the role of the feature, is the BERT embedding. For each role , we can according to and get two lists of feature values of 0 or 1: , . They respectively represent whether the token in the sentence is the start or end of the feature playing the role .

[0082] Denote the loss function of all binary classifiers detecting the start of the feature as , the loss function of detecting the end as , CE is the cross entropy, R is the role set, S is the input sentence, and the formula for getting the loss is:

[0083]

[0084] For and take the mean as the final loss of the feature extractor, and the formula is as follows.

[0085]

[0086] Among them, and are hyperparameters for balancing these three losses.

[0087] In an implementation manner of this embodiment, the initial event extraction template includes an event detection module and an element extraction module. Training the initial event extraction template using the sample text information includes: encoding the input sample text information using the first BERT encoding layer of the event detection module to output event type information and training the event detection module; reading the module parameters and the word vectors of each sentence in the description text; concatenating the word vectors with the vectors corresponding to the event types and inputting them into the second BERT encoding layer of the element extraction module, fixing and randomly initializing it as the module parameters of the element extraction module ; among them, the event detection module includes a first word embedding layer, a first BERT encoding layer, a first BiLSTM layer, and a first CRF layer, and the element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. The second BiLSTM layer receives inputs from the second BERT layer and the first BERT layer of the event detection module simultaneously through horizontal connections, and the first BERT layer and the second BERT layer are connected to each other.

[0088] During the event extraction process, the problem of catastrophic forgetting may occur. The lifelong learning method based on model expansion selected by the present invention realizes knowledge transfer and avoids the forgetting problem. In specific applications, the present invention uses progressive networks, mainly supporting transfer learning between different task sequences through horizontal connections between different neural network layers.

[0089] There are two subtasks in the event extraction model, namely event detection and element extraction. The BERT encoding layer of the event detection module encodes the input sentence, and the relevant parameters are denoted as after training, and the vectors of each token (Tokenization) in the sentence are obtained. When switching to the element extraction module, it concatenates the word vectors of the sentence with the vectors corresponding to the event types as the input. After passing through the BERT encoding layer, the parameters are fixed and its parameters are denoted by random initialization as , the BiLSTM layer of the element extraction module receives inputs simultaneously from its own BERT layer and the BERT layer of the event detection module through horizontal connection, and features are transmitted through the connection between the two BERT layers. The specific formula is as follows:

[0090]

[0091] Among them, is the weight matrix of the th layer of the BERT in the element extraction module, is the horizontal connection from the th layer of the BERT in the event detection module to the th layer of the BERT in the element extraction module, is the input of the th layer of the BERT in the element extraction module. The function is linear: , making the inputs of the middle layers of the BERT model all positive values.

[0092] Through the lifelong learning method, the word features after training the BERT in the event detection module are transferred to the element extraction, enabling it to learn certain prior knowledge and completing the lifelong learning of the model without destroying the original task sequence. Figure 3 is the structural diagram of event extraction based on lifelong learning in the embodiment of the present invention.

[0093] In an implementation manner of this embodiment, after outputting the element role information of the description text, it further includes: in response to a correction instruction, replacing the event type information and the element role information with the correct event type information and the correct element role information; re-inputting the replaced correct event type information, correct element role information, and description text into the element extraction module, outputting the element role information of the description text, and continuing to train the element extraction module.

[0094] During the overall process of event extraction, two types of information biases may occur, namely: classification information in the event detection module and prediction information in the event element extraction module. Since data augmentation is an effective method to improve the robustness of the model, in order to enhance the quality of the data, the present invention audits the incorrect classification information and prediction information through an expert evaluation module, corrects the incorrect data information, and then re-inputs it together with the correct data information into the event element extraction module for training, increasing the amount of training data and also increasing the noisy data, thereby improving the robustness of the model.

[0095] After the event extraction model performs feedback input for event detection and element extraction, in order to improve the generalization ability of the extraction model, it is necessary to update the template in a timely manner, including adding event types and adding event roles in specific event types, etc. Here, the second method is taken as an example. For example, in the "investment" event type of the existing event extraction template, a total of 7 types of event roles are set, namely: "trigger word", "individual investor", "investment organization", "individual investee", "invested organization", "investment amount", and "date". In the example "XX Technology completed a Series XX financing of 50 million yuan, led by XX Capital 1 and followed by XX Capital 2", through event detection, the event type can be extracted as "investment", and through the element extraction module, it can be obtained that: in the event roles, the "investment organization" is "XX Technology", and the "investment amount" is "50 million yuan", but "Series XX", "XX Capital 1" and "XX Capital 2" lack role annotations. Therefore, add event roles: "investment round" is "Series XX", "leading investment organization" is "XX Capital 1", and "following investment organization" is "XX Capital 2". That is, the roles of the "investment" category events in the output updated template are: "trigger word", "individual investor", "investment organization", "individual investee", "invested organization", "investment amount", "date", "investment round", "leading investment organization", and "following investment organization".

[0096] Through the design and update of the event template, the invention continuously improves the adaptability of the model to different data and enhances the accuracy of the event extraction model.

[0097] Taking "On September 25th, the animated image dissemination media platform "Animated Image XX" announced that it has recently completed a Series XX financing of 50 million yuan, led by XX United Creation and followed by XX Youth Creation" as an example, the event detection and element extraction results are shown in Table 3 below:

[0098] Table 3

[0099]

[0100] After expert review, "Animated Image XX" should be "obj-org" (invested organization), which was misidentified as "sub-org" (investment organization). "Series XX financing", "XX United Creation" and "XX Youth Creation" do not have clear corresponding elements in the "investment" category events of the event extraction template. Therefore, add the "number" element as the investment round in the "investment" category events, add the "fol-org" (the second initiator of the event) element in the event extraction template, and add it to the "investment" event extraction elements. Finally, use the correctly adjusted example classification and elements as the input of the event element extraction module for training.

[0101] In the event of "XX Education's news on June 24th, XX Children's Programming completed a tens-of-millions-level Series XX financing at the end of last year. This round of financing was led by XX Capital 3, with XX Capital 4 and XX Venture Capital following up", the event detection and element extraction results are shown in Table 4 below:

[0102] Table 4

[0103]

[0104] The above prediction results have been basically verified by experts and are also retrained through the event element extraction module again.

[0105] Figure 4 This is the event extraction flowchart based on lifelong learning in the embodiments of the present invention. In this embodiment, an event extraction template applicable to relevant event elements including the financial field is constructed. Based on this template, a lifelong learning event extraction method for application model extension is proposed, including two stages: event detection and element extraction. Finally, data error correction and template design update are carried out through expert feedback. It includes constructing an event extraction template for the financial field, an event extraction method based on lifelong learning, and model enhancement based on expert feedback.

[0106] Due to the rapid development of information technology, how to quickly and massively mine the key information of events from text and display it in a structured manner has become an urgent problem to be solved. Therefore, event extraction has emerged. Among them, event extraction in the financial field is of great significance for investors to quickly obtain market information and then make correct decisions. By constructing an event extraction template for financial field event elements to achieve event classification and event element extraction, firstly, it can be well generalized to new event categories. Secondly, it can share the underlying information of event classification and element extraction. At the same time, it can avoid the transmission of incorrect information between tasks. Finally, it can enable the model to simultaneously pay attention to the correspondence between multiple problems and the text, and achieve parallel extraction of event elements.

[0107] By applying the lifelong learning method based on model extension to the event extraction model, the model's memory ability for knowledge is improved, enabling it to continuously integrate new knowledge into the original knowledge during the task training process, so as to maintain a high accuracy rate in new extraction tasks.

[0108] After the event element extraction is completed, the prediction results need to be fed back to the element extraction module to improve the robustness of the model. The present invention mainly enhances the model through two aspects: one is to correct the event elements confirmed as prediction errors after expert review and use them together with the correctly predicted elements as the input of the element extraction module again; the other is to realize the automatic update of the event extraction template through the event type detection and the element extraction results after expert review. The feedback update of data and event templates helps to construct a more complete element template and provides a more reliable guarantee for event extraction.

[0109] In this embodiment, an event extraction model based on event templates in the financial field is designed, and a lifelong learning method based on model extension and model enhancement based on expert feedback are applied, realizing the memory of knowledge by the extraction model and the automatic update of event templates, providing strong support for the accurate extraction of event elements.

[0110] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0111] Embodiment 2

[0112] In this embodiment, a financial field event extraction device based on lifelong learning is also provided to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0113] Figure 5 is a structural block diagram of a financial field event extraction device based on lifelong learning according to an embodiment of the present invention. As Figure 5 shown, the device includes: a configuration module 50, a construction module 52, wherein,

[0114] The configuration module 50 is used to configure an initial event extraction template, wherein the initial event extraction template includes multiple event types and element roles corresponding to each event type;

[0115] The construction module 52 is used to train the initial event extraction template with sample text information and output a target event extraction template, wherein the initial event extraction template includes an event detection module and an element extraction module.

[0116] Optionally, the building block includes: a first training unit configured to input each description text in the sample text information into the event detection module, output event type information of the description text, and train the event detection module; and a second training unit configured to input the event type information and the description text into the element extraction module, output element role information of the description text, and train the element extraction module.

[0117] Optionally, the event detection module includes a first word embedding layer, a first Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a first Bidirectional Long Short-Term Memory (BiLSTM) layer, and a first Conditional Random Field (CRF) layer. The first training unit includes: an input subunit configured to input the description text into the first word embedding layer to obtain a text vector of the description text; an encoding subunit configured to perform embedding encoding on the text vector by using the first BERT encoding layer to obtain a word vector matrix of the description text; a processing subunit configured to input the word vector matrix into the first BiLSTM layer to output a first probability matrix of the description text, where the first probability matrix includes probabilities that each word in the word vector matrix is mapped to each event type label; and an obtaining subunit configured to obtain an optimal tag from the probability matrix by using the first CRF layer and determine the optimal tag as the event type information of the description text.

[0118] Optionally, the element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. The second training unit includes: an input subunit configured to input the description text into the second word embedding layer to obtain an element vector matrix of the description text; an encoding subunit configured to perform embedding encoding on each element vector in the element vector matrix by using the second BERT encoding layer to obtain a role vector matrix of event elements, where the role vector matrix includes element text positions where the event elements belong to each element role; a processing subunit configured to input the role vector matrix into the second BiLSTM layer to output a second probability matrix of the event elements, where the second probability matrix includes probabilities that each event element in the role vector matrix is mapped to each element role label; and an obtaining subunit configured to obtain an optimal tag from the second probability matrix by using the second CRF layer and determine the optimal tag as the element role information of the event elements.

[0119] Optionally, the second BERT encoding layer includes multiple groups of binary classifiers, and the encoding subunit is further configured to: input each vector identification token of the feature vector into each group of binary classifiers respectively, and output the probability values that each identification of the feature vector belongs to the start character and end character of the corresponding feature role of the feature, where each binary classifier corresponds to a feature role.

[0120] Optionally, the construction module includes: an encoding unit, configured to encode the input sample text information by using the first BERT encoding layer of the event detection module, output event type information, and train the event detection module; a reading unit, configured to read the module parameters of the trained event detection module and the word vectors of each sentence in each description text in the sample text information; a processing unit, configured to splice the word vectors with the vectors corresponding to the event type and then input the spliced vectors into the second BERT encoding layer of the feature extraction module, fix the and randomly initialize it as the module parameters of the feature extraction module ; where, the event detection module includes a first word embedding layer, the first BERT encoding layer, a first BiLSTM layer and a first CRF layer, the feature extraction module includes a second word embedding layer, the second BERT encoding layer, a second BiLSTM layer and a second CRF layer, the second BiLSTM layer receives inputs from the second BERT layer and the first BERT layer of the event detection module simultaneously through horizontal connection, and the first BERT layer and the second BERT layer are connected to each other.

[0121] Optionally, the construction module further includes: an error correction unit, configured to, after the second training unit outputs the feature role information of the description text, in response to an error correction instruction, replace the event type information and the feature role information with correct event type information and correct feature role information; an iteration unit, configured to input the replaced correct event type information, the correct feature role information and the description text into the feature extraction module again, output the feature role information of the description text, and continue to train the feature extraction module.

[0122] It should be noted that the above-mentioned each module can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: all the above-mentioned modules are located in the same processor; or, the above-mentioned each module is located in different processors in any combination form.

[0123] Embodiment 3

[0124] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0125] Optionally, in this embodiment, the above storage medium may be configured to store a computer program for executing the following steps:

[0126] S1, configure an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type;

[0127] S2, train the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module.

[0128] Optionally, in this embodiment, the above storage medium may include, but is not limited to, various media that can store computer programs such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0129] An embodiment of the present invention further provides an electronic device including a memory and a processor, where the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

[0130] Optionally, the above electronic device may further include a transmission device and an input / output device, where the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0131] Optionally, in this embodiment, the above processor may be configured to execute the following steps through a computer program:

[0132] S1, configure an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type;

[0133] S2, train the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module.

[0134] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated herein.

[0135] Figure 6It is a structural diagram of an electronic device according to an embodiment of the present invention. As Figure 6 shown, it includes a processor 61, a communication interface 62, a memory 63, and a communication bus 64. Among them, the processor 61, the communication interface 62, and the memory 63 complete mutual communication through the communication bus 64. The memory 63 is used to store a computer program. The processor 61 is used to execute the program stored on the memory 63.

[0136] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0137] In the above embodiments of the present application, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0139] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0141] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs.

[0142] The foregoing are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for extracting financial domain events based on lifelong learning, characterized in that, Including: Configuring an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type; Training the initial event extraction template using sample text information to output a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module; The training of the initial event extraction template using sample text information includes: Encoding the input sample text information using the first BERT encoding layer of the event detection module to output event type information and training the event detection module; Read the module parameters of the trained event detection module and the word vectors of each sentence in each descriptive text in the sample text information; Concatenate the word vector with the vector corresponding to the event type and input it into the second BERT encoding layer of the feature extraction module, and fix the and randomly initialize it as the module parameters of the feature extraction module ; Wherein, the event detection module includes a first word embedding layer, the first BERT encoding layer, a first BiLSTM layer and a first CRF layer, the element extraction module includes a second word embedding layer, the second BERT encoding layer, a second BiLSTM layer and a second CRF layer, the second BiLSTM layer is horizontally connected to the second BERT encoding layer and the first BERT encoding layer of the event detection module to simultaneously receive the inputs of the second BERT encoding layer and the first BERT encoding layer of the event detection module, and the first BERT encoding layer and the second BERT encoding layer are connected to each other to transmit features through the connection and share underlying information.

2. The method according to claim 1, wherein The training of the initial event extraction template using sample text information includes: For each description text in the sample text information, inputting the description text into the event detection module to output the event type information of the description text and training the event detection module; Inputting the event type information and the description text into the element extraction module to output the element role information of the description text and training the element extraction module.

3. The method according to claim 2, characterized in that, The event detection module includes a first word embedding layer, a first Bidirectional Encoder Representations from Transformers (BERT) encoding layer, a first Bidirectional Long Short-Term Memory (BiLSTM) layer and a first Conditional Random Field (CRF) layer. Inputting the description text into the event detection module to output the event type information of the description text includes: Inputting the description text into the first word embedding layer to obtain the text vector of the description text; Performing embedding encoding on the text vector using the first BERT encoding layer to obtain the word vector matrix of the description text; Inputting the word vector matrix into the first BiLSTM layer to output the first probability matrix of the description text, where the first probability matrix includes the probability that each word in the word vector matrix maps to each event type label; Using the first CRF layer to obtain the optimal label from the probability matrix and determining the optimal label as the event type information of the description text.

4. The method according to claim 2, wherein The element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer and a second CRF layer. Inputting the event type information and the description text into the element extraction module to output the element role information of the description text includes: Input the described text into the second word embedding layer to obtain the element vector matrix of the described text; For each element vector in the element vector matrix, use the second BERT encoding layer to perform embedding encoding on the element vector to obtain the role vector matrix of event elements, where the role vector matrix includes the element text positions where the event elements belong to each element role; Input the role vector matrix into the second BiLSTM layer to output the second probability matrix of the event elements, where the second probability matrix includes the probabilities of each event element in the role vector matrix mapping to each element role label; Use the second CRF layer to obtain the optimal label from the second probability matrix and determine the optimal label as the element role information of the event elements.

5. The method according to claim 4, characterized in that The second BERT encoding layer includes multiple groups of binary classifiers. Using the second BERT encoding layer to perform embedding encoding on the element vector to obtain the role vector matrix of event elements includes: Input each vector identifier token of the element vector into each group of binary classifiers respectively, and output the probability values of each identifier of the element vector belonging to the starting character and the ending character of the corresponding element role, where each binary classifier corresponds to an element role.

6. The method according to claim 2, characterized in that, After outputting the element role information of the described text, the method further includes: In response to the error correction instruction, replace the event type information and the element role information with the correct event type information and the correct element role information; Input the replaced correct event type information, the correct element role information and the described text into the element extraction module again, output the element role information of the described text, and continue to train the element extraction module.

7. An event extraction device in the financial field based on lifelong learning, characterized in that, Including: A configuration module for configuring an initial event extraction template, where the initial event extraction template includes multiple event types and element roles corresponding to each event type; A construction module for training the initial event extraction template with sample text information and outputting a target event extraction template, where the initial event extraction template includes an event detection module and an element extraction module; The training of the initial event extraction template with sample text information includes: Using the first BERT encoding layer of the event detection module to encode the input sample text information and output event type information to train the event detection module; Read the module parameters of the trained event detection module and the word vectors of each sentence in each description text of the sample text information; Concatenate the word vector with the vector corresponding to the event type and input it into the second BERT encoding layer of the feature extraction module, fixing the and randomly initialize it as the module parameters of the feature extraction module ; Among them, the event detection module includes a first word embedding layer, the first BERT encoding layer, a first BiLSTM layer, and a first CRF layer. The element extraction module includes a second word embedding layer, a second BERT encoding layer, a second BiLSTM layer, and a second CRF layer. The second BiLSTM layer is horizontally connected to the second BERT encoding layer and the first BERT encoding layer of the event detection module to simultaneously receive the inputs of the second BERT encoding layer and the first BERT encoding layer of the event detection module. The first BERT encoding layer and the second BERT encoding layer are interconnected to transfer features through the connection and share underlying information.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program, when running, executes the method steps described in any one of claims 1 to 6 above.

9. An electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, communication interface, and memory complete mutual communication through a communication bus; wherein: The memory is used to store a computer program; The processor is used to execute the method steps described in any one of claims 1 to 6 by running the program stored on the memory.

Images