Gas accident entity relation joint extraction method based on improved CasRel

Abstract

The invention discloses a gas accident entity relation joint extraction method based on improved CasRel. The method comprises the following steps: firstly, acquiring a gas accident case text, and carrying out entity relationship labeling on text data by utilizing a Brt labeling tool; inputting the labeled data into a BERT pre-training model to obtain word vectors, and introducing adversarial training to optimize model parameters; inputting the word vectors into a head entity tagging device, and identifying all possible head entities in the sentences; and forming a head entity relation tuple through a Self-Cross-Self-Attention mechanism, and inputting the head entity relation tuple into a series of specific relation tail entity annotators to complete relation triple extraction. According to the method, the situation that overlapped triads and nested triads cannot be completely extracted through traditional relation extraction is overcome, the accuracy rate and coverage rate of relation extraction are greatly improved, and reliable data support is provided for construction of a gas accident knowledge graph.

Description

Technical Field

[0001] The invention relates to a gas accident entity relationship joint extraction method based on improved CasRel, belonging to the technical field of text entity relationship extraction. Background Art

[0002] In recent years, with the continuous growth of urban gas consumption, the frequency and complexity of gas accidents have put forward higher requirements for urban safety and emergency response. Gas accident information usually exists in various accident reports, news reports and regulatory documents in the form of unstructured text. How to accurately extract key information (such as accident cause, accident location, accident type and damage) from these text data has become a key step in building a gas accident knowledge graph.

[0003] Information extraction is the first step in building a knowledge graph and is the key technical foundation. Information extraction includes two tasks: named entity recognition and relationship extraction. The main task of named entity recognition is to identify specific words and technical terms in unstructured data, while relationship extraction is to extract the semantic relationship between entities, and finally obtain the triple composed of entity relationships, that is, (head entity, relationship, tail entity). The traditional information extraction method is a pipeline learning method, which divides it into two independent tasks. The disadvantage of this method is that it is easy to cause information transmission errors and information loss. With the deepening of research, the joint modeling of entity recognition and relationship extraction tasks can achieve better results. However, some joint extraction models are not suitable for text extraction in the field of gas accidents, and the entity density in gas accident texts is high. There are many cases where multiple relationships in the same sentence share the same entity. In order to solve these problems. The above will cause difficulties in the entity relationship extraction of gas accident texts, resulting in poor model extraction effect. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a gas accident entity relationship joint extraction method based on improved CasRel in view of the deficiencies in the prior art, which can effectively perform entity relationship joint extraction on gas accident case data.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A gas accident entity relationship joint extraction method based on improved CasRel, characterized in that the calculation method includes the following steps:

[0007] S1: Obtain gas accident investigation reports and gas accident case texts, and use the Brat annotation tool to annotate the text data with entity relationships before model training;

[0008] S2: When using the pre-trained BERT model for feature extraction, the model begins to encode the input text and obtains the encoded word vector containing semantic information;

[0009] S3: Introduce adversarial training to adjust the parameters of the model to improve the generalization ability of the model. This adversarial training is applied to the feature extraction process of the model during the training phase, thereby affecting the final word vector representation;

[0010] S4: Input the word vector into the head entity tagger and use the proximity principle to identify all possible head entities;

[0011] S5: Introducing the "Self-Cross-Self-Attention" attention mechanism, using the self-attention mechanism to capture the dependencies between words in the input sequence, enhancing the model's understanding of a single entity and its related context; then using the cross-attention mechanism to fuse the features of different entities; finally, the features processed by the cross-attention process are subjected to another layer of self-attention operation to further integrate the previously obtained information and enhance the system's understanding of the global context;

[0012] S6: Finally, through the specific relationship tail entity decoder, all possible preset relationships and tail entities are identified, the triple information contained in the input data is obtained, and the joint extraction of gas accident emergency response entity relationships is completed.

[0013] The present invention provides a joint extraction method of gas accident entity relations based on improved CasRel, which replaces the usual pipeline entity relation extraction binary classification method with a new relation extraction method integrating the Attention mechanism relation mapping function, thereby overcoming the problem that the traditional relation extraction cannot completely extract overlapping triples and nested triples; adversarial training is introduced, and adversarial samples are added during the training process, that is, a slight perturbation is made to the input data, so that the model can learn resistance to adversarial disturbances; at the same time, the pre-training model we use has a strong generalization ability to obtain word vectors, and can fully describe the word-level and sentence-level feature relations in the gas accident text. The pre-training model is combined with the relation mapping function to greatly improve the accuracy and coverage of relation extraction.

[0014] Beneficial effects:

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention introduces adversarial training. By adding adversarial samples during the training process, that is, making small disturbances to the input data, the model can learn resistance to disturbances and maintain good performance when facing noise data or input changes, thereby enhancing the model's adaptability to uncertainty. The present invention introduces the "Self-Cross-Self-Attention" attention mechanism, using the self-attention mechanism to capture the dependencies between words in the input sequence, enhancing the model's understanding of a single entity and its related context; then through the cross-attention mechanism, the features of different entities are fused; finally, the features after the cross-attention processing are subjected to another layer of self-attention operation to further integrate the previously obtained information and strengthen the system's understanding of the global context. The present invention improves the CasRel cascade tagging framework model, effectively solving the problem of multiple relationships sharing the same entity in the same sentence of the gas accident text, and completing the extraction of triples in the gas accident text. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a specific implementation method of a gas accident triple extraction method;

[0018] Figure 2 This is a model framework diagram of the gas accident entity relationship joint extraction method of the improved CasRel of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] Example:

[0021] A gas accident entity relationship joint extraction method based on improved CasRel, the specific calculation method includes the following steps:

[0022] S1: Obtain gas accident cases and investigation reports, use annotation tools to annotate the entity relationships of text data before model training, and divide the text into test data sets and training and validation data sets after preprocessing;

[0023] S2: When using the pre-trained BERT model for feature extraction, the model begins to encode the input text and obtains the encoded word vector containing semantic information;

[0024] S3: Introduce adversarial training to adjust the model parameters to improve the generalization ability of the model. This adversarial training is applied to the feature extraction process of the model during the training phase, thereby affecting the final word vector representation;

[0025] S4: Input the word vector into the head entity tagger and use the proximity principle to identify all possible head entities;

[0026] S5: The introduction of the "Self-Cross-Self-Attention" attention mechanism enables the model to efficiently process multiple entities in the text, deeply explore their inherent relevance, and ultimately improve the ability to accurately locate and identify multiple entities in the scene;

[0027] S6: Finally, through the specific relationship tail entity decoder, all possible preset relationships and tail entities are identified, the triple information contained in the input data is obtained, and the joint extraction of gas accident emergency response entity relationships is completed.

[0028] The above-mentioned improved CasRel gas accident entity relationship joint extraction method obtains gas accident case text, and uses the Brat annotation tool to annotate the text data with entity relationships before model training. First, the annotated data is input into the BERT pre-training model, and the feature extraction of the sentence is performed to obtain the encoded word vector containing semantic information. Adversarial training is introduced to adjust the parameters of the model to improve the generalization ability of the model. The obtained word vector is passed through the head entity decoder to find all possible head entities in the sentence, and then the Self-Cross-Self-Attention mechanism is introduced to assign different weights to different entities, and extract vectors that can better represent the entity. Finally, through the specific relationship tail entity decoder, all possible preset relationships and tail entities are identified to complete the joint extraction of gas accident entity relationships and obtain the triple information contained in the input data.

[0029] In step S1, gas accident cases and investigation report texts are obtained, and the relevant text data is crawled by the lightweight crawler framework Scrapy. After the text data is collected, the collected text is cleaned and deduplicated. The obtained gas accident data set is divided into a training set and a test set in a ratio of 7:3. The training set is annotated with the Brat annotation tool to annotate the data entities and relationships, and the entity categories are defined as: accident name, accident time, accident location, accident type, accident impact, accident cause, emergency subject, and emergency task. According to the entity information, the entity relationship type is set to: the time of occurrence is, the location of occurrence is, belongs to, the impact is, the cause is, and the emergency measure is.

[0030] In step S2, the BERT pre-trained model is suitable for constructing word vectors for gas accident texts, and the model outputs word vectors in units of characters, overcoming the defects of traditional Chinese word segmentation relationship extraction methods. The word vectors obtained through the BERT model have stronger generalization capabilities and can fully describe the feature relationships between character level, word level, and sentence level.

[0031] In step S3, the adversarial training adjusts the parameters of the model and is applied to the feature extraction process of the model during the training phase, thereby affecting the final word vector representation; first, adversarial training can improve the robustness of the model. By adding adversarial samples during the training process, that is, making slight perturbations to the input data, the model can learn to resist the perturbations, which enables the model to maintain good performance when facing noisy data or input changes, thereby enhancing the model's adaptability to uncertainty; secondly, adversarial training helps alleviate the overfitting problem. After the introduction of adversarial training, the model is forced to optimize on the perturbed input data. This additional "difficulty" prompts the model to explore the feature space more extensively, thereby reducing excessive reliance on specific patterns in the training data; in addition, adversarial training can enhance the model's ability to learn fine-grained features; in the task of relation extraction, subtle semantic differences may significantly affect the correct classification of relations. Through adversarial training, the model can pay better attention to these details, improve the ability to distinguish complex relations and fuzzy boundaries, and ultimately improve the accuracy of relation extraction.

[0032] In step S4, the head entity tagger is used to identify all possible head entities. The word vector output in step S3 is input into the head entity tagger to identify all potential head entities in the text. The start and end of the head entity are represented by 1, and other non-head entity words are represented by 0. For multiple head entity detection, the proximity principle is used for judgment: all words between the starting position of a head entity (marked as 1) and its nearest end position (also marked as 1) constitute the head entity. The specific calculation formula of the head entity tagger is as follows:

[0033]

[0034] in, and Respectively represent the input sequence h n The probability that the i-th entity is recognized as the starting and ending position in the corresponding ith entity. When the probability exceeds the preset threshold, the value of the position is set to 1, otherwise it is set to 0. The purpose is to avoid extracting too many entities and affecting the experimental results. σ() is the sigmoid activation function, W start and W end is the trainable weight, b start and b end Indicates the bias value, m i It represents the encoding of the i-th token in the input sequence. When there are multiple head entities in a sentence, the scope of the entity is determined by the matching principle of the closest start and end position tokens, and this matching method ensures the integrity of the entity.

[0035] In step S5, the "Self-Cross-Self-Attention" attention mechanism first captures the dependencies between words in each input sequence by applying self-attention to each sequence, thereby enhancing the model's understanding of a single entity and its related context. Through this operation, the system can obtain a fine-grained representation of each entity in the original text without losing local information. Next, through the cross-attention mechanism, the model can fuse the features of different entities. Finally, the features processed by the cross-attention are subjected to another layer of self-attention operation to further integrate the previously obtained information and enhance the system's understanding of the global context. The calculation formula of the self-attention function is as follows:

[0036]

[0037] Among them, Q, K, and V represent the query matrix, key matrix, and value matrix respectively. The three are connected through the vector of each entity and the parameter matrix W Q , W K , W V Then the self-attention function is used to calculate the attention weight of each position, and finally the output is generated through linear change. The attention function used in the present invention is the dot product attention calculation method to speed up the operation and save memory space, while introducing the scaling factor To balance the effect of excessive dot product value, the output is expressed as H1:

[0038] H1=SelfAttn(X)

[0039] Furthermore, the cross attention calculation is performed on the output H1 and another input sequence Y to further extract context information. The calculation formula of the cross attention is as follows:

[0040]

[0041] Among them, Q H1 represents the query matrix generated for H1, which is obtained by combining H1 with the weight matrix W Q Multiply them together to get, which represents the query information for each position in H1, K Y represents the key matrix generated for the sequence Y, which is obtained by adding Y to the weight matrix W K Multiply them together to get the feature information of each position in Y, V Y Represents the value matrix generated for the sequence Y, which is obtained by dividing Y by the weight matrix W V Multiply them together to get the value of K. Y The corresponding actual feature information, d K Represents the dimension of the key vector, which is used for scaling when calculating the attention weight to prevent the dot product value from being too large;

[0042] Furthermore, cross attention extracts richer feature representations through the relationship between H1 and Y, and generates a new representation H2:

[0043] H2=CrossAttn(H1,Y)

[0044] Furthermore, self-attention is applied again to the output H2 to capture higher-level contextual information extracted from the cross attention, thereby further strengthening the representation. The output of this step is:

[0045] H3=SelfAttn(H2)

[0046] In this step, the model can not only capture complex contextual dependencies through the self-attention mechanism, but also integrate the interaction characteristics between multiple entities to improve the final entity recognition and location matching accuracy.

[0047] In step S6, the specific relationship tail entity decoder is consistent with the head entity annotator in terms of architecture, and is composed of a group of object pointer markers, which also follow the proximity rule; for each head entity Traverse all the tail entities related to it. If the corresponding tail entity is not found, set all the tail entity label positions to 0; the probability calculation formula of the i-th word as the start and end position of the tail entity is as follows:

[0048]

[0049] in, and denote the probability of the i-th token in the input sequence being recognized as the start and end position of the tail entity, respectively, and x i is the encoding representation of the i-th character in the input sentence. When the calculated result p exceeds a certain threshold, it is considered to be a boundary and marked as 1, otherwise it is 0. and is the training parameter matrix under the relationship condition, and is the bias vector under the relation condition, and σ() is the sigmoid activation function. Given a sentence representation x, the likelihood function under the optimization of the head entity tagger determines the range of the head entity o as shown below:

[0050]

[0051] Where L is the length of the sentence, I is the indicator function, which is 1 if the internal expression is correct, otherwise it is 0. and The binary markers representing the start and end positions of the i-th character tail entity in sentence x, respectively. They represent the starting weight, starting bias, ending weight, and ending bias respectively.

[0052] The present invention provides a joint extraction method of gas accident entity relations based on improved CasRel, which replaces the usual pipeline entity relation extraction binary classification method with a new relation extraction method integrating the Attention mechanism relation mapping function, thereby overcoming the problem that the traditional relation extraction cannot completely extract overlapping triples and nested triples; adversarial training is introduced, and adversarial samples are added during the training process, that is, a slight perturbation is made to the input data, so that the model can learn resistance to adversarial disturbances; at the same time, the pre-training model we use has a strong generalization ability to obtain word vectors, and can fully describe the word-level and sentence-level feature relations in the gas accident text. The pre-training model is combined with the relation mapping function to greatly improve the accuracy and coverage of relation extraction.

Claims

1. A gas accident entity relationship joint extraction method based on improved CasRel, characterized in that: The following steps are involved: S1: Obtain gas accident cases and investigation reports, use annotation tools to annotate the entity relationships of text data before model training, and divide the text into test data sets and training and validation data sets after preprocessing; S2: When using the pre-trained BERT model for feature extraction, the model begins to encode the input text and obtains the encoded word vector containing semantic information; S3: Introduce adversarial training to adjust the model parameters to improve the generalization ability of the model. This adversarial training is applied to the feature extraction process of the model during the training phase, thereby affecting the final word vector representation; S4: Input the word vector into the head entity tagger and use the proximity principle to identify all possible head entities; S5: The "Self-Cross-Self-Attention" attention mechanism is introduced to enable the model to efficiently process multiple entities in the text, deeply explore their inherent relevance, and ultimately improve the ability to accurately locate and identify multiple entities in the scene; S6: Finally, through the specific relationship tail entity decoder, all possible preset relationships and tail entities are identified, the triple information contained in the input data is obtained, and the joint extraction of gas accident emergency response entity relationships is completed.

2. According to claim 1, a gas accident entity relationship joint extraction method based on improved CasRel is characterized in that: In step S1, gas accident cases and investigation report texts are obtained, and relevant text data are crawled by using the lightweight crawler framework Scrapy. After the text data is collected, the collected text is cleaned and deduplicated, and 3348 gas accident case data are obtained. The data set is divided into a training set and a test set in a ratio of 7:

3. The training set is annotated with the Brat annotation tool to annotate the data entities and relationships, and the entity categories are defined as: accident name, accident time, accident location, accident type, accident impact, accident cause, emergency subject, and emergency task. According to the entity information, the entity relationship type is set to: the time of occurrence is, the location of occurrence is, belongs to, the impact is, the cause is, and the emergency measure is.

3. The construction method according to claim 1, characterized in that: The BERT pre-trained model in step S2 is suitable for constructing word vectors for gas accident texts, and the model outputs word vectors in units of characters, overcoming the defects of traditional Chinese word segmentation relationship extraction methods. The word vectors obtained through the BERT model have stronger generalization capabilities and can fully describe the feature relationships between character level, word level, and sentence level.

4. The construction method according to claim 1, characterized in that: The step S3 introduces adversarial training to adjust the parameters of the model, which is applied to the feature extraction process of the model during the training phase, thereby affecting the final word vector representation; first, adversarial training can improve the robustness of the model. By adding adversarial samples during the training process, that is, making slight perturbations to the input data, the model can learn resistance to adversarial perturbations, which enables the model to maintain good performance when facing noisy data or input changes, thereby enhancing the model's ability to adapt to uncertainty; Secondly, adversarial training helps alleviate the overfitting problem. After the introduction of adversarial training, the model is forced to optimize on the perturbed input data. This extra "difficulty" prompts the model to explore the feature space more extensively, thereby reducing over-reliance on specific patterns in the training data. In addition, adversarial training can enhance the model's ability to learn fine-grained features. In the task of relation extraction, subtle semantic differences may significantly affect the correct classification of relations. Through adversarial training, the model can pay better attention to these details, improve the ability to distinguish complex relations and fuzzy boundaries, and ultimately improve the accuracy of relation extraction.

5. The construction method according to claim 1, characterized in that: The step S4 uses a head entity tagger to identify all possible head entities, and inputs the word vector outputted from step S3 into the head entity tagger to identify all potential head entities in the text; the start and end of the head entity are both represented by 1, and other non-head entity words are represented by 0; for multiple head entity detections, the proximity principle is used for determination: all words between a head entity start position (marked as 1) and its nearest end position (also marked as 1) constitute the head entity; the specific calculation formula of the head entity tagger is as follows: in, and Respectively represent the input sequence h n The probability that the i-th entity is recognized as the starting and ending position in . When the probability exceeds the preset threshold, the value of the position is set to 1, otherwise it is set to 0. The purpose is to avoid extracting too many entities and affecting the experimental results. σ( ) is the sigmoid activation function, W start and W end is the trainable weight, b start and b end Indicates the bias value, m i It represents the encoding of the i-th token in the input sequence. When there are multiple head entities in a sentence, the scope of the entity is determined by the matching principle of the closest start and end position tokens, and this matching method ensures the integrity of the entity.

6. The construction method according to claim 1, characterized in that: The step S5 uses the "Self-Cross-Self-Attention" attention mechanism, which first captures the dependencies between words in each input sequence by applying self-attention to each sequence, thereby enhancing the model's understanding of a single entity and its related context. Through this operation, the system can obtain a fine-grained representation of each entity in the original text without losing local information. Next, through the cross-attention mechanism, the model can fuse the features of different entities. Finally, the features after the cross-attention processing are subjected to a layer of self-attention operation to further integrate the previously obtained information and strengthen the system's understanding of the global context; the calculation formula of the self-attention function is as follows: Among them, Q, K, and V represent the query matrix, key matrix, and value matrix respectively. The three are connected through the vector of each entity and the parameter matrix W Q , W K , W V Then the self-attention function is used to calculate the attention weight of each position, and finally the output is generated through linear change; the attention function used in the present invention is the dot product attention calculation method to speed up the operation and save memory space, and introduce the scaling factor To balance the effect of excessive dot product value, the output is expressed as H1: H1=SelfAttn(X) The cross attention calculation is performed on the output H1 and another input sequence Y to further extract context information. The calculation formula of the cross attention is as follows: Among them, Q H1 represents the query matrix generated for H1, which is obtained by combining H1 with the weight matrix W Q Multiply them together to get, which represents the query information for each position in H1, K Y represents the key matrix generated for the sequence Y, which is obtained by adding Y to the weight matrix W K Multiply them together to get the feature information of each position in Y, V Y Represents the value matrix generated for the sequence Y, which is obtained by dividing Y by the weight matrix W V Multiply them together to get the value of K. Y The corresponding actual feature information, d K Represents the dimension of the key vector, which is used for scaling when calculating the attention weight to prevent the dot product value from being too large; Here, cross attention extracts richer feature representations through the relationship between H1 and Y, and generates a new representation H2: H2=CrossAttn(H1,Y) Self-attention is applied again to the output H2 to capture higher-level contextual information extracted from the cross attention to further strengthen the representation. The output of this step is: H3=SelfAttn(H2) In this step, the model can not only capture complex contextual dependencies through the self-attention mechanism, but also integrate the interaction characteristics between multiple entities to improve the final entity recognition and location matching accuracy.

7. The construction method according to claim 1, characterized in that: The architecture of the specific relation tail entity decoder in step S6 is consistent with that of the head entity annotator, and is composed of a group of object pointer markers, which also follow the proximity rule; for each head entity Traverse all the tail entities related to it. If the corresponding tail entity is not found, set all the tail entity label positions to 0; the probability calculation formula of the i-th word as the start and end position of the tail entity is as follows: in, and denote the probability of the i-th token in the input sequence being recognized as the start and end position of the tail entity, respectively, and x i is the encoding representation of the i-th character in the input sentence. When the calculated result p exceeds a certain threshold, it is considered to be a boundary and marked as 1, otherwise it is 0. and is the training parameter matrix under the relationship condition, and is the bias vector under the relational condition, σ() is the sigmoid activation function; given a sentence representation x, the likelihood function under the optimization of the head entity tagger determines the range of the head entity o as shown below: Where L is the length of the sentence, I is the indicator function, which is 1 if the internal expression is correct, otherwise it is 0. and The binary markers representing the start and end positions of the i-th character tail entity in sentence x, respectively. They represent the starting weight, starting bias, ending weight, and ending bias respectively.

Images