A general event relationship recognition method based on neural network

Abstract

The present invention discloses a general event relationship recognition method based on a neural network, the recognition method comprising: obtaining entity elements, events composed of elements, and event relationship pairs through training data; checking whether the values ​​of the same element type in an event pair are the same, counting the distribution variance of the consistency ratio in different event relationship types, and then selecting according to the size of the variance, selecting several with larger variances as classification elements; for a text to be classified, obtaining event relationship pairs to be classified; according to the event relationship pairs to be classified, marking the entities in the original text of the text to be classified with indicators according to the classification elements; and using the marked original text as input of a classification neural network.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a general event relationship recognition method based on a neural network. Background Art

[0002] Among the tasks related to natural language processing, named entity recognition and classification are two basic tasks, which usually directly use character / word level information in the text as input. There are currently relatively mature solutions for these two types of tasks, and they have achieved good results in different application scenarios.

[0003] Based on the results of named entity recognition, the task of entity relationship recognition is derived, such as Figure 1 As shown in Figure 1, it can be applied to the task of combining knowledge graph elements or events. The input of such tasks usually requires some transformation of the text or the addition of entity indication information so that the model can perceive the specific entities to be concerned.

[0004] Based on the combination of entities, we can get a higher level of representation - event. An event can be represented as a combination of multiple entities, which can be recorded as ("element 1", "element 2", ...), for example ("subject", "action", "object"). Further refinement of multiple events with similar logic to entities can obtain event relationships, such as Figure 1 shown).

[0005] Event relations are usually represented as tuples of event pairs and their relations, i.e. ("event 1", "event 2", "event relation"). Event 1 and event 2 constitute an event pair. In an event pair, the former is the preceding event and the latter is the following event. Common event relations include causal relations, sequential relations, and parallel relations. The event relation recognition task is closer to the knowledge that people obtain from texts and is the basis for constructing higher-level knowledge such as logical reasoning.

[0006] Existing event relationship recognition is mostly processed based on traditional solutions for trigger word recognition. This solution usually involves people first sorting out a series of correspondence tables between trigger words and corresponding event relationships based on experience, such as {"therefore":"causal relationship", "and":"parallel relationship"}, and then searching for relationship trigger keywords that appear in the text, and matching the corresponding event relationships based on the trigger words. The solution based on trigger words has many defects. On the one hand, the solution cannot handle scenarios with more than two events well. When there are more than two events, there is a lack of better methods to determine which two events the relationship obtained through the trigger words exists between. On the other hand, the solution faces the problem of difficulty in keyword sorting. Since the trigger words need to be sorted manually, it will bring more manual work. Furthermore, the solution ignores rich semantic information. For some scenarios, even if the trigger words appear, it does not mean that there is a corresponding relationship between the events. At this time, the event relationship judgment must be provided by text other than the trigger words. In addition, some methods have attempted to combine event elements with certain fixed features of the text to obtain some structured representations, such as whether certain elements appear or not, whether the elements of two events are consistent, the distance between elements, and other information, and then classify them based on traditional machine learning algorithms (such as decision trees, logistic regression, etc.). This method also has the disadvantage of not making full use of the contextual semantics, and requires manual feature construction, which lacks flexibility. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a general event relationship recognition method based on a neural network. Some embodiments of the present invention can integrate event information and original text using labels, so that the event relationship classification task can be converted into an ordinary text classification task for processing. The event element indicator added to the original text integrates the position information of the element in the text, the event information to which it belongs, and the semantic information of the element. After obtaining the text representation with the indicator, the attention network is used to extract information, and then a unified text representation is obtained, and finally the text classification is performed based on the representation.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A general event relationship recognition method based on a neural network, the recognition method comprising: obtaining entity elements, events composed of elements, and event relationship pairs through training data; checking whether the values ​​of the same element type in an event pair are the same, counting the distribution variance of the consistency ratio in different event relationship types, and then selecting according to the size of the variance, selecting several with larger variances as classification elements; for a text to be classified, obtaining event relationship pairs to be classified; according to the event relationship pairs to be classified, marking the entities in the original text of the text to be classified with indicators according to the classification elements; and using the marked original text as input of a classification neural network.

[0010] Preferably, the event relationship pair includes a category indicating that there is no relationship between the event pair.

[0011] Preferably, marking the entities in the original text to be classified with indicators according to the classification elements according to the event relationship pairs to be classified includes: for the same text, if multiple parallel event relationship pairs are generated, it is necessary to generate multiple paragraphs marked with indicators in one-to-one correspondence.

[0012] Preferably, the classification neural network includes: adding a position encoding represented by a sin / cos function, obtaining the Q, K, and V representations of each word / embedding input, and then performing attention calculation with other words / words in the text. Through the attention calculation, the word / word at each position is integrated with the context information, and then a CNN layer is attached on this basis to integrate the results of the entire text to obtain a representation that can represent the entire text; finally, a fully connected layer is used to compress the text vector into several category labels, and the category label on each data is obtained by using softmax calculation, and then the cross entropy loss is calculated with the true label, and the back propagation algorithm is used for optimization, and continuous iteration is performed to obtain the optimal model.

[0013] Preferably, the classification neural network outputs the most likely category of the event relationship pair and its score.

[0014] Preferably, a score segmentation threshold is selected according to the output result, the output is further selected based on the threshold, and the result after the second selection is used as the final result.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. Integrating events and original texts as model input by adding tags to the original text, fusing event information with original text information; 2. Using Attention combined with CNN to extract text semantic information; 3. Using the variance of the consistent distribution of elements in event pairs as a method for element selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 Schematic diagram of background technology.

[0018] Figure 2 Schematic diagram of the process of the embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] There are still many problems in event relationship identification:

[0021] 1. How to integrate event information and text information?

[0022] As mentioned above, the use of trigger-based or manually constructed features has the problem of insufficient use of context semantics. When judging the relationship of event granularity, context is obviously crucial, and context information is retained in the original text. Therefore, it is impossible to discard the original text if you want to obtain complete context information. Obtaining a better combination of event pairs and text has become a key issue in event relationship recognition.

[0023] The event itself also contains a lot of information: the position of the entity element in the text, the value of the entity itself, the event to which it belongs, etc. The difference in this information will lead to different event relationship results. Different entity words first constitute an important distinction. "Orange" and "Beijing" have completely different information. Even the same entity word may bring different information according to the difference in its position in the text. For example, the word "apple" usually refers to fruit if "eat" is closer in the context, and if "mobile phone" is closer in the context, it is more likely to refer to an Apple phone. In addition, entity words with the same attributes of different events should be distinguishable. For example, if event 1 and event 2 in a text have attributes such as subject, action, and object, the subject of event 1 should be able to correspond to other attributes such as object and action in the same event, and distinguish the difference between its action and object and event 2. Different information will obviously lead to different recognition results.

[0024] 2. How to use the model to extract text information?

[0025] Common text information extraction modules include recurrent neural network (RNN), convolutional neural network (CNN) and the newly emerged attention module. Each module has its own advantages and disadvantages when processing text tasks. After adding many gate switches, the RNN model can selectively memorize information. Generally speaking, it has a good feature extraction effect, but it may face the problem of long-term memory loss, and because RNN can only process in series, the processing speed is usually slower. CNN is mostly used in image-related tasks. After adaptive transformation, it can also be used in text-related tasks. The convolution structure enables it to better pay attention to the fine-grained information in the text. Its parallel processing can greatly accelerate the model training / prediction speed, but due to its natural convolution processing characteristics, the long-distance dependency information between texts cannot be well extracted. The newly emerged attention module has received great attention after the release of Bert because it can naturally process long text sequences, extract distant dependencies, and is very suitable for parallel processing. It has been widely used in recent tasks, but the native attention module lacks the perception of the position information of words / words in the text, and usually requires additional position encoding information. How to choose the right module to suit your task is a difficult point.

[0026] 3. When there are many entity attributes, how to choose the key attributes?

[0027] In some event definitions, there may be many elements. For example, a typical event may have attributes such as subject, action, object, time, and location. These attributes do not all contribute equally to the distinction of event relationships. For example, when defining causal relationships, time, location, etc. are obviously not as important as actions and objects in distinguishing event relationships. Therefore, in event definitions with more attributes, it is difficult to distinguish which elements are "useful" for event relationship identification. If all elements are treated equally without distinction, it may bring a lot of noise information.

[0028] In view of the above problem, a method is proposed, which comprises the following steps:

[0029] 1. Prepare training data for event relationship classification

[0030] The training data for this step usually comes from manual annotation. A simplified example sample is as follows:

[0031]

[0032]

[0033] A sample contains the original "text", and the "event" list contains event tuples marked from the text. There is a sequence between these events, and their indexes correspond to those in the "relationship". The first digit in the relationship triple represents the index of the preceding event, index 0 represents the first event, index 1 represents the second event, and so on. The second digit represents the index of the following event, and its notation is the same as the preceding event. The third digit represents the relationship between the two events. According to the above example, it means that there is a "causal relationship" between the second event and the first event. If the relationship between the two events is not in the previously defined target event relationship set, it is usually not listed.

[0034] 2. Supplement negative samples

[0035] The training data usually only marks the event pairs with target relations in the text, while ignoring other possible event relation combinations that are not in the target event relation set. Considering the application scenario, the model needs to judge the event pairs formed by any two events and output the scores. If the classification only has the target event relation, the classification of those event pairs that are not in the target event relation set is prone to errors, because at this time the model needs to "hard" stuff them into an event relation category that is actually incorrect. Therefore, for this scenario, it is necessary to add a non-existent category, such as "unknown", so that the model will not overfit when learning, and will classify all event relations into existing relations. The model can correctly distinguish event pairs that do not have non-existent relations. As shown below, there is a causal relationship between event 1 and event 0, but the relationship between event 0 and event 1 is not in the defined target event relation set, so it is marked as "unknown".

[0036]

[0037]

[0038] 3. Select entity types that contribute to event relationship classification

[0039] The logic of this selection method is: if there is a large difference in the distribution of an element in different event relationship pairs, then the element is more likely to be used to assist in the classification of event relationships. For example, in the above example, the subject of multiple event pairs is "Zhang San", then the relationship between the two events will be more dependent on contextual information and information elements such as "action" and "object". In specific operations, you can check whether the values ​​of the same element type in an event pair are the same, and then count the distribution variance of this consistency ratio in different event relationship types, and then select according to the size of the variance, and select the top N elements with larger variances, such as the top three, which are the elements that are more important for event relationship classification.

[0040] Further examples include:

[0041] Action: {'sequential': 0.11301044634377967, 'intention': 0.3241525423728814, 'parallel': 0.9796954314720813}, standard deviation: 0.369

[0042] Time: {'sequence': 0.9971509971509972, 'intention': 0.9957627118644068, 'parallel': 1.0}, standard deviation: 0.002.

[0043] Among them, the consistent ratio of element values ​​between two event pairs corresponding to the relationship of succession in the entire training data for the action element is 0.11301044634377967, while the consistent ratio of element values ​​between two event pairs corresponding to the relationship of intention is 0.3241525423728814.

[0044] 4. Construct text data with event information

[0045] Special marks are used in the original text to indicate the event information, and "<>" is used as an indicator for the event entity. Specifically, label indicators are added to distinguish between the preceding event and the following event. The preceding event uses " <h>< / h> " (head), post events use " <t>< / t> "(tail). In addition, for different types of entities, additional distinctions are added. For example, for subject, action, and object, indicators s, a, and o are added respectively, forming " <hs>< / hs> "," <ha>< / ha> "," <ho>< / ho> ". For some scenarios, the same entity appears in both the preceding event and the following event. To avoid label nesting, a separate label is used to indicate this situation, such as " " (union) to indicate that the entity appears in both the preceding event and the succeeding event. After the above processing, the sample data will generate labeled text in the following form, where the former event relationship is unknown and the latter event relationship label is "causal relationship":

[0046] <us> Zhang San< / us> <ha> Go< / ha> <ho> Beijing< / ho> <ta> join< / ta> <to> Meeting< / to> Then we went to Quanjude to eat roast duck.

[0047] <us> Zhang San< / us> <ta> Go< / ta> <to> Beijing< / to> <ha> join< / ha> <ho> Meeting< / ho> Then we went to Quanjude to eat roast duck.

[0048] As shown above: Although the original input is the same text, different indicator markers are added to obtain different inputs for the model.

[0049] 5. Generate category labels

[0050] For different types of event relations, the id of each event relation is obtained through dictionary mapping, for example, {"causal relation": 1, "sequential relation": 2, "unknown": 3}, which is used as the target variable of the classification model, i.e., y, to convert the task into a conventional text multi-classification task. Since the final task is converted into a text classification task, the model can be used to solve general event relation classification tasks, not just limited to single event relation judgment.

[0051] 6. Perform text preprocessing, including optional normalization and word segmentation operations;

[0052] 7. Get embedding

[0053] After preprocessing, the characters / words need to be converted into numerical representations, that is, tokenization. At this time, the representation obtained is relatively discrete, and it is impossible to obtain expressions with semantic information. It is necessary to enrich the semantic information through model learning or with the help of external data, such as using word vectors obtained by BERT pre-training, or directly using an embedding layer to learn semantic representations from existing data when there is enough training data.

[0054] 8. Use the attention + CNN model for iterative learning

[0055] The attention model can well learn the associations between words in long texts. Therefore, for texts using special tags, it can pay attention to the associations between annotations used to indicate events in the text. However, since attention itself lacks direct position information encoding, a position encoding represented by a sin / cos function is added with reference to the processing logic of "Attention is all you need". For each word / embedding input, its Q, K, and V representations are obtained, and then attention calculations are performed with other words / words in the text. Through attention calculations, the words / words at each position are integrated with the context information, and then a CNN layer is attached on this basis to integrate the results of the entire text to obtain a representation that can represent the entire text.

[0056] Finally, a fully connected layer is used to compress the text vector into several category labels. The category label on each data is obtained by using softmax calculation, and then the cross entropy loss is calculated with the true label. The back propagation algorithm is used for optimization and continuous iteration is performed to obtain the optimal model.

[0057] 9. Prediction stage

[0058] Event pairs are obtained through a common method. The entities selected in step 3 are used to construct the text to be classified according to step 4. Then, the text is preprocessed according to the same operation as step 6. After that, it is input into the model for classification judgment. The model will output the maximum possible category and its score for the event pair. At this point, the score segmentation threshold (default 0.5) can be selected based on the model verification results, and the model output can be further selected based on the threshold. The result after the second selection is used as the final result.

[0059] Although the above embodiments have been described in detail for the present invention, it should be understood by those skilled in the art that modifications or improvements may be made based on the contents disclosed in the present invention without departing from the spirit and scope of the present invention, and such modifications and improvements are within the spirit and scope of the present invention.

Claims

1. A general event relationship recognition method based on neural network, characterized in that: The identification method comprises: Obtain entity elements, events composed of elements, and event relationship pairs through training data; Check whether the values ​​of the same feature type in an event pair are the same, count the distribution variance of this consistency ratio in different event relationship types, and then select according to the variance size, select the first N features with larger variance as classification features, and select them after arranging them in order of variance size, where N ≥ 3, N is a positive integer; For the text to be classified, obtain the event relationship pair to be classified; According to the relationship between the events to be classified, entities in the original text to be classified are marked with indicators according to the classification elements; Use "<>" as an indicator for event entities; specifically, label indicators are added to distinguish between pre-events and post-events, and pre-events use " <h>< / h> " (head), post-event uses " <t>< / t> "(tail); In addition, for different types of entities, additional distinctions are made, and for subject, action, and object, indicators s, a, and o are added respectively, forming " <hs>< / hs> ”、" <ha>< / ha> ”、" <ho>< / ho> "; for some scenarios, the same entity appears in both the preceding event and the following event. To avoid label nesting, a separate label is used to indicate this situation, using " "(union)" to indicate that the entity appears in both the preceding and succeeding events; The marked original text is used as the input of the classification neural network.

2. The neural network-based general event relationship identification method according to claim 1, characterized in that: The event relationship pair includes a category indicating that there is no relationship between the event pair.

3. The neural network-based general event relationship recognition method according to claim 1, characterized in that: The step of marking the entities in the original text to be classified with indicators according to the classification elements according to the event relationship pairs to be classified includes: for the same text, if multiple parallel event relationship pairs are generated, multiple one-to-one corresponding paragraphs marked with indicators need to be generated.

4. The neural network-based universal event relationship identification method according to claim 1, characterized in that: The classification neural network comprises: Add a position encoding represented by a sin / cos function. For each word / embedding of the input, obtain its Q, K, V representation, and then perform attention calculation with other words / words in the text. Through attention calculation, the word / word at each position is integrated with the context information. Then, add a CNN layer on this basis to integrate the results of the entire text and obtain a representation that can represent the entire text. Finally, a fully connected layer is used to compress the text vector into several category labels. The category label on each data is obtained by using softmax calculation, and then the cross entropy loss is calculated with the true label. The back propagation algorithm is used for optimization and continuous iteration is performed to obtain the optimal model.

5. The neural network-based universal event relationship identification method according to claim 1, characterized in that: The classification neural network outputs the most likely category and score of the event relationship pair.

6. The neural network-based universal event relationship recognition method according to claim 5, characterized in that: A score segmentation threshold is selected based on the output result, and the output is further selected based on the threshold, and the result after the second selection is taken as the final result.

Images