Aviation accident text data extraction method based on cyclic BERT and corner marking

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to an aviation accident text data extraction method based on loop BERT and corner marking, which comprises the following steps: S1, identifying a trigger word of aviation accident report text data by using a double-loop BERT model; s2, jointly calculating a text data argument vector of the aviation accident report by using a classifier model; s3, calculating the total confidence coefficient of the argument vector of the text data according to the vector of the trigger word of the text data and the argument label; s4, decoding the event triple of the aviation accident report text data by using the angle mark matrix to obtain a total loss value of the text data; and S5, judging and extracting aviation accident report text data according to the total loss value. According to the method, context information and argument positions are identified through a double-loop training model, then argument confidence is calculated in a combined mode, redundant errors and cascade errors are reduced, the triple prediction accuracy is increased, model complexity is reduced by using an angle marking method, and extraction result comprehensiveness is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to an aviation accident text data extraction method based on cyclic BERT and corner marking. Background Art

[0002] In the process of aviation accident investigation and analysis, the aviation accident report text covers the process, cause and description of related events of the accident. The large amount of information contained is crucial for accident investigation and safety improvement. However, these texts are usually stored in discrete unstructured forms, showing the characteristics of complex structure and huge amount of information, which makes it difficult to establish strong associations, limit data analysis, and make it difficult to deeply understand the root causes and potential risks of accidents. Therefore, it is urgent to use effective information extraction technology to accurately and efficiently extract key event information from unstructured aviation accident texts.

[0003] Knowledge extraction technology based on deep learning uses deep learning models to identify entities, relations, attributes, events and other information in texts, and stores them in the form of (subject, relation, object) triples (SRO) to achieve structured representation of text information, which can be used to build knowledge graphs, semantic understanding and reasoning, and data mining and analysis. Traditional information extraction usually adopts a pipeline method, which is divided into two independent subtasks: first named entity recognition (NER) and then relation extraction (RE). Although this method is flexible in modeling, it has problems such as error accumulation, lack of information interaction between subtasks, information redundancy, and difficulty in extracting long dependencies. With the development of deep learning technology, scholars have proposed joint extraction technology based on deep learning, which aims to use a single model to extract entities and relations at the same time to solve the problems of pipeline methods. For example, CasRel (Entity Relation Extraction Model Based on Joint Decoding) is an end-to-end binary annotation framework that solves the problem of entity pair discretization by mapping the subject to the object function. TPLinker (Single-step Entity-Relation Joint Extraction Model Based on Entity Linking) is a single-stage relation extraction model that solves the joint extraction of entity relations through token pair links. PRGC (Joint Relation Triple Extraction Based on Latent Relations and Global Correspondence) uses relation-specific sequence annotations to handle overlapping entities and reduces redundancy by predicting latent relations. OneRel (Single-module Single-step Entity-relation Joint Extraction Model) uses corner labeling to achieve single-module single-step extraction of entities and relations.

[0004] Although these models have shown good performance in general domain datasets, there are still challenges in extracting textual knowledge in the aviation accident field. The main reason is that there are a lot of potential logic and entity pair overlaps in aviation accident report texts, and there is a lack of targeted knowledge extraction models. Therefore, in order to effectively extract structured data from aviation accident report texts, it is necessary to develop knowledge extraction models suitable for this field. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides an aviation accident text data extraction method based on cyclic BERT and corner marking. The double-loop pre-training model can accurately identify context information and argument positions, and then reduce redundant errors and cascade errors by jointly calculating argument confidences, increase the accuracy of triplet predictions, and use the corner marking method to reduce model complexity and increase the comprehensiveness and accuracy of the extraction results.

[0006] To achieve the above object, the present invention discloses the following technical solution:

[0007] A method for extracting aviation accident text data based on cyclic BERT and corner tagging, comprising:

[0008] S1: Using a dual-loop BERT model to identify trigger words in aviation accident report text data;

[0009] The text data of aviation accident reports is obtained, and the text data feature vector is embedded. The embedded vector of the text data is encoded into a label sequence, and the text data trigger word position is embedded and annotated; the feature vector of the position embedding is processed by the BERT model for secondary encoding, and the argument vector is identified to complete the text data trigger word training, specifically:

[0010]

[0011] in, is the vector of the trigger word in the text data; softmax is the function mapping relationship; W trigger is the weight of the context information vector of the trigger word in the text data; h i is the vector of the context information of the i-th trigger word; b trigger is the vector deviation of the text data trigger word; i is the aviation accident report text data number;

[0012] S2: Use the classifier model to jointly calculate the argument vector of the text data of aviation accident reports;

[0013] Use double-loop BERT to jointly represent feature vectors, calculate the probability of text data argument vectors through the classifier model, and combine the text data with the argument head word probability and the probability of the last word of the joint argument Combining, the predicted text data meta-tag is:

[0014]

[0015] in, is the i-th argument vector of the predicted text data; is the probability of the head word of the ith argument; is the probability of the tail word of the ith argument;

[0016] S3: Calculate the total confidence CN of the text data argument vector according to the text data trigger word vector in step S1 and the text data argument label in step S2:

[0017]

[0018] Among them, CN is the total confidence of the text data argument vector; Π is the product operation; Confidence is the confidence calculation function mapping;

[0019] S4: using the corner labeling matrix to decode the aviation accident event triples of the aviation accident report text data to obtain the total loss value of the text data;

[0020] S41: The argument vector with the highest total confidence CN of the text data argument vector in step S3 is stored in the corner labeling matrix R for decoding feature labeling; the aviation accident event triplet feature value of the text data is calculated using the corner labeling matrix assignment function:

[0021]

[0022] in, is the eigenvalue of the aviation accident event triple of the text data; R is the corner label matrix of the aviation accident event triple; is the vector of the subject word in the aviation accident event triple; O is the vector of the object word in the aviation accident event triple; is the cyclic calculation of subject words and object words; φ is the ReLU activation function mapping; W is the weight parameter for classifier model training; is the concatenation operation of the subject word vector and the object word vector; b is the bias parameter of the classifier model training;

[0023] S42: Predict the loss value L based on the trigger word trigger , argument prediction loss value L arg And the corner mark prediction loss value L rel The weighted sum is used to obtain the total loss value L of the text data. total for:

[0024] L total =αLtrigger +βL arg +γL rel ;

[0025] Among them, L total is the total loss value of the text data; L trigger is the trigger word prediction loss value; L arg is the loss value of the argument prediction; L rel is the prediction loss value of the corner mark; α is the trigger word prediction parameter; β is the argument prediction parameter; γ is the corner mark prediction parameter;

[0026] S5: For the aviation accident event triplet feature value of the text data in step S41, set the error output threshold; when the total loss value L of the text data output in step S42 total When the error is greater than or equal to the error output threshold, the weight matrix parameters and bias item parameters of the classifier model in step S2 are updated using the back propagation algorithm, and the execution is continued by returning to step S2; when the total loss value L of the text data output in step S42 is total When it is lower than the error output threshold, the aviation accident event triplet in step S41 is output as the extracted aviation accident report text data.

[0027] Preferably, in step S1, text data feature vector embedding is performed, the embedding vector of the text data is encoded into a label sequence, and the text data trigger word position embedding annotation is performed, specifically as follows:

[0028] S11: embedding the text data of the aviation accident report into a feature vector; embedding the text data into entity words, segmentation embedding and position embedding to form the embedding vector of the text data as follows:

[0029] x i =e i +p i +s i ;

[0030] Among them, x i is the embedding vector of text data; e i is the entity word embedding vector of text data; p i is the segment embedding vector of text data; i Embedding vectors for the positions of text data;

[0031] S12: Encode the embedding vector of the text data in step S11 into a label sequence to implement the training of entity words, trigger words and context information in the text data of the aviation accident report:

[0032] h i =BERT(x i );

[0033] Among them, h i is the vector of the context information of the i-th trigger word; BERT is the multi-layer Transformer hidden function mapping;

[0034] S13: The trigger word position embedding annotation of the text data of the aviation accident report; when BERT is used for processing for the first time, the position embedding mark in the text data is 0, and after processing with multiple layers of hidden units, the trigger words and trigger word positions are identified; when BERT is used for processing for the second time, the trigger word position embedding mark in the text data is 1.

[0035] Preferably, in step S2, a double-loop BERT is used to jointly represent the feature vectors, and the probability of the text data argument vector is calculated by the classifier model, specifically:

[0036] S21: Use the double-loop BERT model to jointly represent the feature vectors, and obtain the joint feature vector of the text data argument vector as follows:

[0037] J i ={B1(w1+e1),…,B i (w i +e i ),…B m (w m +e m )};

[0038] Among them, J i is the i-th joint feature vector of the text data argument vector; B i is the i-th embedding vector of the context information of the text data; w i is the i-th word vector of the text data; m is the total number of joint feature vectors of the text data argument vectors;

[0039] S22: The probability of determining the text data argument vector through the classifier model is:

[0040]

[0041] in, Combine the argument head word probability for text data; Combine the argument tail word probability for text data; The weight of the head word of the text data argument; is the weight of the tail word of the text data argument; J s is the joint feature vector of the head word of the text data argument; J e is the joint feature vector of the tail words of the text data argument; B is the embedding vector of the context information of the text data; w is the word vector of the text data; e is the embedding vector of the entity word of the text data.

[0042] Preferably, the decoding feature marks in step S41 include: a first decoding feature mark HB-TB, a second decoding feature mark HB-TE, a third decoding feature mark HE-TE, a fourth decoding feature mark Minus and a fifth decoding feature mark Null;

[0043] The first decoding feature mark HB-TB is decoded from the character corresponding to the start position of the subject word to the character corresponding to the start position of the object word;

[0044] The second decoding feature mark HB-TE is decoded from the character corresponding to the start position of the subject word to the character corresponding to the end position of the object word;

[0045] The third decoding feature mark HE-TE is decoded from the character corresponding to the end position of the subject word to the character corresponding to the end position of the object word;

[0046] The fourth decoding feature mark Minus is to decode in sequence according to the subword position order;

[0047] The fifth decoding feature flag Null is a word that does not need to be decoded and output.

[0048] Preferably, the trigger word prediction loss value L in step S42 is trigger , the trigger word loss value is calculated by the cross entropy loss function:

[0049]

[0050] Among them, L trigger is the prediction loss value of the trigger word; N is the number of trigger word samples; C is the number of trigger word categories; is the real label of the trigger word; is the prediction probability of the trigger word; c is the category number of the trigger word.

[0051] Preferably, the argument prediction loss value L in step S42 is arg , specifically:

[0052] The argument prediction loss model is used to measure aviation accident event arguments. The equations for the predicted loss value at the beginning of the argument and the predicted loss value at the end of the argument are as follows:

[0053]

[0054] in, Predict the loss value for the starting position of the argument; Predict the loss value for the end position of the argument; is the real label of the starting position of the argument; Predict the probability for the starting position of the argument; is the actual label of the end position of the argument; Predict the probability of the end position of the argument;

[0055] Combining the predicted loss value of the starting position and the predicted loss value of the ending position of each argument, the total predicted loss value of the aviation accident event argument is obtained as follows:

[0056]

[0057] Among them, A is the set of all arguments; L arg Predict the loss value for the argument.

[0058] Preferably, the corner mark prediction loss value L in step S42 is rel Specifically, the corner label matrix prediction loss is used to measure the difference between the corner label matrix extracted by the double-loop BERT model and the actual result. The specific formula is:

[0059]

[0060] Among them, L rel is the prediction loss value of the corner mark; K is the number of relationship categories; is the true label of the k-th category relationship between the i-th entity and the j-th entity; is the predicted probability of the corner labeling matrix; k is the relationship category number.

[0061] Preferably, the error output threshold is set in step S5, specifically:

[0062] S511: Calculate the error term of the output layer of the classifier model:

[0063]

[0064] Among them, δ(L total ) is the error term of the output layer of the classifier model; is the total loss function L total The weighted input z(L total )’s partial derivative; is the predicted probability of the classifier model; y is the true label of the classifier model;

[0065] S512: Calculate the error term of the hidden layer of the classifier model:

[0066]

[0067] Among them, δ(l) is the error term of the lth layer of the classifier model; W(l+1) is the weight of the l+1th layer of the classifier model; δ(l+1) is the error value of the l+1th layer of the classifier model; is the element-wise multiplication calculation; g′(z(l)) is the derivative of the activation function;

[0068] S513: Construct the error output threshold Δ as:

[0069]

[0070] Among them, Δ is the error output threshold, which is specifically the error ratio between the loss function difference of the current layer and the output layer and the loss function value of the output layer. When Δ≥50%, the current loss function error does not meet the requirements, and gradient calculation is required to update the classifier model parameters.

[0071] Preferably, in step S5, the weight matrix parameters and bias item parameters of the classifier model in step S2 are updated using a back propagation algorithm, specifically:

[0072] S521: Calculate the total loss function L total The gradient is:

[0073]

[0074]

[0075] in, is the loss function L total The partial derivative of the weight matrix W(l) of the lth layer; W(l) is the weight matrix of the lth layer of the classifier model; a(l-1) is the activation value of the (l-1)th layer of the classifier model, representing the output of the neurons in this layer; is the loss function L total The partial derivative of the bias term b(l) of the lth layer; b(l) is the bias of the lth layer of the classifier model;

[0076] S522: Update the classifier model parameters using the gradient descent optimization algorithm:

[0077]

[0078] Among them, W(l) is the weight matrix of the lth layer of the classifier model; b(l) is the bias of the lth layer of the classifier model; α is the set learning rate, which is used to control the step size of parameter update.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] (1) Based on the traditional event extraction method, the present invention proposes a corner-labeled joint event extraction method based on double-loop BERT to extract key information from aviation accident report texts and solve the cascade error and redundant error problems existing in step-by-step event extraction. The method is used to extract key information such as accident phenomena, accident causes, accident results, and main objects from aviation accident report texts.

[0081] (2) The present invention utilizes double-loop BERT to identify and classify trigger words and locate arguments, combines context information and feature vectors to jointly calculate argument confidence, and uses the corner marking method to jointly decode event information, thereby enhancing information relevance and thereby improving the accuracy of text information extraction.

[0082] (3) The present invention improves the corner marking strategy in the traditional model. On the basis of being able to solve the problem of entity pair overlap, it solves the problem of implicit logical relationships in the same argument and the problem of entity pair overlap, thereby improving the ability to process aviation accident information. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a control block diagram of the aviation accident text data extraction method based on cyclic BERT and corner marking of the present invention;

[0084] Figure 2 Schematic diagram of the dual-loop BERT model architecture of the present invention;

[0085] Figure 3 Schematic diagram of the embedding layer representation of the double-loop BERT model of the present invention;

[0086] Figure 4 This is an example diagram of the corner marking method of the present invention;

[0087] Figure 5 This is an example diagram of the optimized corner marking method of the present invention. DETAILED DESCRIPTION

[0088] The exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0089] The embodiment of the present invention provides a method for extracting aviation accident text data based on cyclic BERT and corner marking, such as Figure 1As shown, a double-loop BERT model is used to identify trigger words of aviation accident report text data; a classifier model is used to jointly calculate the text data argument vector of aviation accident report; the total confidence of the text data argument vector is calculated according to the vector of the text data trigger word and the argument label; the event triples of the aviation accident report text data are decoded using a corner labeling matrix to obtain the total loss value of the text data; and the aviation accident report text data is extracted according to the total loss value; which includes:

[0090] Step S1: Use the double-loop BERT model to identify the trigger words of the aviation accident report text data. Obtain the text data of the aviation accident report, embed the text data feature vector, encode the text data embedding vector into a label sequence, and embed the text data trigger word position annotation, specifically:

[0091] Step S11: embedding the text data of the aviation accident report into a feature vector; embedding the text data into entity words, segmentation and position, and forming the embedding vector of the text data as follows:

[0092] x i =e i +p i +s i ;

[0093] Among them, x i is the embedding vector of text data; e i is the entity word embedding vector of text data; p i is the segment embedding vector of text data; i Embedding vector for the position of text data.

[0094] Step S12: Encode the embedding vector of the text data in step S11 into a label sequence to implement the training of entity words, trigger words and context information in the text data of the aviation accident report:

[0095] h i =BERT(x i );

[0096] Among them, h i is the vector of the context information of the i-th trigger word; BERT is a multi-layer Transformer hidden function mapping.

[0097] Step S13: embedding and marking the trigger word position of the text data of the aviation accident report; when BERT is used for processing for the first time, the position embedding mark in the text data is 0, and after processing by multiple layers of hidden units, the trigger words and trigger word positions are identified; when BERT is used for processing for the second time, the position embedding mark of the trigger word in the text data is 1.

[0098] Step S14: The feature vector embedded in the position is subjected to secondary encoding processing by the BERT model to identify the argument vector and complete the text data trigger word training, specifically:

[0099]

[0100] in, is the vector of the trigger word in the text data; softmax is the function mapping relationship; W trigger is the weight of the context information vector of the trigger word in the text data; h i is the vector of the context information of the i-th trigger word; b trigger is the vector deviation of the trigger word in the text data; i is the number of the aviation accident report text data.

[0101] like Figure 2 The figure shows a schematic diagram of the architecture of the corner-marked joint event extraction model based on double-loop BERT of the present invention; the overall architecture of the model is explained. The model is divided into three modules: encoding layer, prediction layer and decoding layer. The encoding layer uses the BERT pre-trained model to encode, convert the input sentence into a feature vector, and encode the feature vector into a label sequence through multiple layers of hidden units. The prediction layer is responsible for the prediction of trigger words and arguments, and uses the BERT pre-trained model to identify entities and trigger words. Through the double-loop BERT method, trigger words are used to locate arguments and learn contextual information. Through the activation function in the classifier, combined with entities, trigger words and semantic information, the confidence is jointly calculated and the position of the argument with the highest confidence is marked, and stored in the corner-marking matrix in the decoding layer. The decoding layer uses the corner-marking method to combine and score all possible argument pairs through the corner-marking matrix of the predefined relationship, and outputs the combination with the highest score as structured triple data.

[0102] like Figure 3 The figure shows a schematic diagram of the embedding layer representation of the double-loop BERT model of the present invention; it illustrates that the sentence input into the BERT model is encoded into a feature vector through word embedding, segment embedding, and position embedding. Word embedding assigns a feature vector to each subword and symbol in the sentence, and adds a "[CLS]" symbol at the beginning of the input sentence for classification tasks, and inserts a "[SEP]" symbol at the end of each sentence for sentence segmentation; segment embedding classifies and marks subwords in the same segment, and subwords in the same segment are marked with the same symbol; position embedding marks the position information of each subword and symbol, and the initial position embedding value is 0.

[0103] Step S2: Use the classifier model to jointly calculate the text data argument vector of the aviation accident report.

[0104] Step S21: Use double-loop BERT to jointly represent the feature vectors, and calculate the text data argument vector through the classifier model:

[0105] J i ={B1(w1+e1),…,B i (w i +e i ),…B m (w m +e m )};

[0106] Among them, J i is the i-th joint feature vector of the text data argument vector; B i is the i-th embedding vector of the context information of the text data; w i is the i-th word vector of the text data; m is the total number of joint feature vectors of the text data argument vectors.

[0107] Step S22: Determine the probability of the text data argument vector through the classifier model:

[0108]

[0109] in, Combine the argument head word probability for text data; Combine the argument tail word probability for text data; The weight of the head word of the text data argument; is the weight of the tail word of the text data argument; J s is the joint feature vector of the head word of the text data argument; J e is the joint feature vector of the tail words of the text data argument; B is the embedding vector of the context information of the text data; w is the word vector of the text data; e is the embedding vector of the entity word of the text data.

[0110] Step S23: Combine the text data with the argument head word probability and the probability of the last word of the joint argument Combining, the predicted text data meta-tag is:

[0111]

[0112] in, is the i-th argument vector of the predicted text data; is the probability of the head word of the ith argument; is the probability of the tail word of the ith argument.

[0113] Step S3: Calculate the total confidence CN of the text data argument vector according to the text data trigger word vector in step S1 and the text data argument label in step S2:

[0114]

[0115] Among them, CN is the total confidence of the text data argument vector; Π is the product operation; Confidence is the confidence calculation function mapping.

[0116] Step S4: Use the corner marker matrix to decode the aviation accident event triples of the aviation accident report text data to obtain the total loss value of the text data.

[0117] Step S41: Store the argument vector with the highest total confidence CN of the argument vector of the text data in step S3 in the corner marker matrix R for decoding feature marking, specifically including: the first decoding feature marker HB-TB is decoded from the character corresponding to the starting position of the subject word to the character corresponding to the starting position of the object word; the second decoding feature marker HB-TE is decoded from the character corresponding to the starting position of the subject word to the character corresponding to the ending position of the object word; the third decoding feature marker HE-TE is decoded from the character corresponding to the ending position of the subject word to the character corresponding to the ending position of the object word; the fourth decoding feature marker Minus is decoded in sequence according to the sub-word position order; the fifth decoding feature marker Null is a character that does not need to be decoded and output.

[0118] The corner label matrix assignment function is used to calculate the aviation accident event triple eigenvalues ​​of the text data:

[0119]

[0120] in, is the eigenvalue of the aviation accident event triple of the text data; R is the corner label matrix of the aviation accident event triple; is the vector of the subject word in the aviation accident event triple; O is the vector of the object word in the aviation accident event triple; is the cyclic calculation of subject words and object words; φ is the ReLU activation function mapping; W is the weight parameter for classifier model training; is the concatenation operation of the subject word vector and the object word vector; b is the bias parameter for classifier model training.

[0121] like Figure 4The following is an example diagram of the angle marking method of the present invention; taking "foreign object causes blade damage" as an example, in the matrix with the predefined relationship of "causes", for the "HB-TB" marking position, the word "foreign object" corresponding to the row is the start of the subject word, and the word "blade" corresponding to the column is the start of the object word. By traversing row by row, the triple (foreign object, causes, blade) is parsed; for the "HB-TE" marking position, the word "foreign object" corresponding to the row is the start of the subject word, and the word "damage" corresponding to the column is the end of the object word. By traversing row by row, the triple (foreign object, causes, blade damage) is parsed; for the "HE-TE" marking position, the word "object" corresponding to the row is the end of the subject word, and the word "damage" corresponding to the column is the end of the object word. By traversing row by row, the triple (foreign object, causes, blade damage) is parsed.

[0122] If there is an implicit logical relationship in the same argument, for example, in the event of "blade fatigue damage", there is a logical relationship of (blade damage, cause, blade fatigue), then when decoding the subject "blade damage", the word "fatigue" needs to be skipped. Therefore, on the basis of the original four marking type labels ("HB-TB", "HB-TE", "HE-TE" and "-"), the "Null" marking is added, indicating that the word at this position is not recognized and the decoding continues by skipping this word. This marking usually appears in the middle part, as Figure 3 shown. The "Null" marking is different from the "-" marking. The former is defined as a key marking during decoding. If this marking is detected, the information at this position will be skipped and the decoding will continue backward, while the latter is not recognized during decoding.

[0123] such as Figure 5The following is an example diagram of the optimized corner marking method of the present invention, which illustrates the optimized corner marking method example adopted when processing sentences with implicit logical relationships in the same argument. Taking the event of "the blade is damaged due to fatigue" as an example, in this event, there is a logical relationship of (blade damage, cause, blade fatigue). When decoding the main body word "blade damage", several words such as "due to", "fatigue", "produce", and "generate" need to be skipped. When decoding the object word "blade fatigue", the word "due to" needs to be skipped. Therefore, when traversing line by line, for the "Null" mark after the recognized "HB-TB" mark, skip the word "due to" corresponding to this mark column; for the "Null" mark after the recognized "HB-TE" mark, skip the words "due to", "fatigue", "produce", and "generate" corresponding to each mark row respectively. Based on this, in the matrix with the predefined relationship of "cause", for the "HB-TB" mark position, the word "blade" in the row is the start of the main body word, and the word "blade" in the column is the start of the object word. By traversing line by line, a triple (blade, cause, blade) is parsed; for the "HB-TE" mark position, the word "many" in the row is the start of the main body word, and the word "damage" in the column is the end of the object word. By traversing line by line, a triple (many, cause, blade damage) is parsed; for the "HE-TE" mark position, the word "foreign object" in the row is the end of the main body word, and the word "damage" in the column is the end of the object word. By traversing line by line, a triple (foreign object, cause, blade damage) is parsed.

[0124] Step S42: Calculate the trigger word loss value L through the cross-entropy loss function trigger which is:

[0125]

[0126] where L trigger is the predicted loss value of the trigger word; N is the number of trigger word samples; C is the number of trigger word categories; is the true label of the trigger word; is the predicted probability of the trigger word; c is the trigger word category number.

[0127] The equations for measuring the aviation accident event arguments, the predicted loss value of the argument start position, and the predicted loss value of the argument end position using the argument prediction loss model are:

[0128]

[0129] where is the predicted loss value of the argument start position; is the predicted loss value of the argument end position; is the true label of the argument start position; is the predicted probability of the argument start position; is the actual label of the end position of the argument; Predict the probability of where the argument ends.

[0130] Combining the predicted loss value of the starting position and the predicted loss value of the ending position of each argument, the total predicted loss value of the aviation accident event argument is obtained as follows:

[0131]

[0132] Among them, A is the set of all arguments; L arg Predict the loss value for the argument.

[0133] The corner label matrix prediction loss is used to measure the difference between the corner label matrix extracted by the double-loop BERT model and the actual result. The corner label prediction loss value L rel The specific formula is:

[0134]

[0135] Among them, L rel is the prediction loss value of the corner mark; K is the number of relationship categories; is the true label of the k-th category relationship between the i-th entity and the j-th entity; is the predicted probability of the corner labeling matrix; k is the relationship category number.

[0136] Predict the loss value L based on the trigger word trigger , argument prediction loss value L arg And the corner mark prediction loss value L rel The weighted sum is used to obtain the total loss value L of the text data. total for:

[0137] L total =αL trigger +βL arg +γL rel ;

[0138] Among them, L total is the total loss value of the text data; L trigger is the trigger word prediction loss value; L arg is the loss value of the argument prediction; L rel is the corner tag prediction loss value; α is the trigger word prediction parameter; β is the argument prediction parameter; γ is the corner tag prediction parameter.

[0139] S5: For the aviation accident event triplet feature value of the text data in step S41, set the error output threshold; calculate the classifier model output layer error term as:

[0140]

[0141] Among them, δ(L total ) is the error term of the output layer of the classifier model; is the total loss function L total The weighted input z(L total )’s partial derivative; is the predicted probability of the classifier model; y is the true label of the classifier model.

[0142] The error term of the hidden layer of the classifier model is calculated as:

[0143]

[0144] Among them, δ(l) is the error term of the lth layer of the classifier model; W(l+1) is the weight of the l+1th layer of the classifier model; δ(l+1) is the error value of the l+1th layer of the classifier model; is the element-wise multiplication calculation; g′(z(l)) is the derivative of the activation function.

[0145] The build error output threshold is:

[0146]

[0147] Among them, Δ is the error ratio between the loss function difference of the current layer and the output layer and the loss function value of the output layer. When Δ≥50%, the current loss function error does not meet the requirements, and gradient calculation is required to update the classifier model parameters.

[0148] When the total loss value L of the text data output in step S42 total When it is greater than or equal to the error output threshold, the back propagation algorithm is used to update the weight matrix parameters and bias item parameters of the classifier model in step S2, and the execution is continued by returning to step S2; the total loss function L is calculated. total The gradient is:

[0149]

[0150] in, is the loss function L total The partial derivative of the weight matrix W(l) of the lth layer; W(l) is the weight matrix of the lth layer of the classifier model; a(l-1) is the activation value of the (l-1)th layer of the classifier model, representing the output of the neurons in this layer; is the loss function L total The partial derivative of the bias term b(l) of the lth layer; b(l) is the bias of the lth layer of the classifier model.

[0151] The classifier model parameters are updated using the gradient descent optimization algorithm:

[0152]

[0153] Among them, W(l) is the weight matrix of the lth layer of the classifier model; b(l) is the bias of the lth layer of the classifier model; α is the set learning rate, which is used to control the step size of parameter update.

[0154] When the total loss value L of the text data output in step S42 total When it is lower than the error output threshold, the aviation accident event triplet in step S41 is output as the extracted aviation accident report text data.

[0155] The aviation accident report text data obtained by the embodiment of the present invention is:

[0156] "text":"A fatigue crack in the No. 1 main landing gear axle was not detected. The fatigue crack was caused by a corroded hole in the brake system connection flange.","entities":["No. 1 main landing gear","axle","fatigue crack","brake system","flange","corroded hole"], "triggers":["not detected","connection","corrosion","caused"]. "arguments":["A fatigue crack in the No. 1 main landing gear axle was not detected","fatigue crack was caused by a corroded hole in the brake system connection flange"]. "triples"[["Axle","Presence","Fatigue Crack"],["Flange","Presence","Corroded Hole"],["Fatigue Crack","Located","Axle"],["Corroded Hole","Located","Flange"],["Axle","Located","No.1 Main Landing Gear"],["Hole","Occurrence","Corrosion"],["Axle","Occurrence","Fatigue Crack Undetected"],["Fatigue Crack","Cause of Occurrence","A corroded hole in the brake system connection flange"].

[0157] The embodiments of the present invention have the following beneficial effects: In order to extract key accident information from the text of aviation accident reports and solve the problems of cascade errors and redundant errors in step-by-step fault event extraction, the embodiments of the present invention propose a corner-marked joint event extraction method based on double-loop BERT, which is used to extract key information such as accident phenomena, accident causes, accident results, and main objects from the text of aviation accident reports. The double-loop BERT model is used to identify and classify trigger words and locate arguments, and the argument confidence is jointly calculated in combination with context information and feature vectors. The corner-marking method is used to jointly decode event information, enhance information relevance, and thus improve the accuracy of text information extraction. On the basis of solving the problem of entity pair overlap, it can solve the problem of implicit logical relationships in the same argument and the problem of entity pair overlap, thereby improving the processing capability of aviation accident information. The analysis of actual cases can prove that the present invention can successfully obtain accident information.

[0158] The embodiments described above are only descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for extracting aviation accident text data based on cyclic BERT and corner tagging, characterized in that , which includes: S1: Using a dual-loop BERT model to identify trigger words in aviation accident report text data; The text data of aviation accident reports is obtained, and the text data feature vector is embedded. The embedded vector of the text data is encoded into a label sequence, and the text data trigger word position is embedded and annotated; the feature vector of the position embedding is processed by the BERT model for secondary encoding, and the argument vector is identified to complete the text data trigger word training, specifically: in, is the vector of the trigger word in the text data; softmax is the function mapping relationship; W trigger is the weight of the context information vector of the trigger word in the text data; h i is the vector of the context information of the i-th trigger word; b trigger is the vector deviation of the text data trigger word; i is the aviation accident report text data number; S2: Use the classifier model to jointly calculate the argument vector of the text data of aviation accident reports; Use double-loop BERT to jointly represent feature vectors, calculate the probability of text data argument vectors through the classifier model, and combine the text data with the argument head word probability and the probability of the last word of the joint argument Combining, the predicted text data meta-tag is: in, is the i-th argument vector of the predicted text data; is the probability of the head word of the ith argument; is the probability of the tail word of the ith argument; S3: Calculate the total confidence CN of the text data argument vector according to the vector of the text data trigger word in step S1 and the text data argument label in step S2: Among them, CN is the total confidence of the text data argument vector; Π is the product operation; Confidence is the confidence calculation function mapping; S4: using the corner labeling matrix to decode the aviation accident event triples of the aviation accident report text data to obtain the total loss value of the text data; S41: The argument vector with the highest total confidence CN of the text data argument vector in step S3 is stored in the corner labeling matrix R for decoding feature labeling; the aviation accident event triplet feature value of the text data is calculated using the corner labeling matrix assignment function: in, is the eigenvalue of the aviation accident event triple of the text data; R is the corner label matrix of the aviation accident event triple; is the vector of the subject word in the aviation accident event triple; O is the vector of the object word in the aviation accident event triple; is the cyclic calculation of subject words and object words; φ is the ReLU activation function mapping; W is the weight parameter for classifier model training; is the concatenation operation of the subject word vector and the object word vector; b is the bias parameter of the classifier model training; S42: Predict the loss value L based on the trigger word trigger , argument prediction loss value L arg and the corner mark prediction loss value L rel The weighted sum is used to obtain the total loss value L of the text data. total for: L total =αL trigger +βL arg +γL rel Among them, L total is the total loss value of the text data; L trigger is the trigger word prediction loss value; L arg is the loss value of the argument prediction; L rel is the prediction loss value of the corner mark; α is the trigger word prediction parameter; β is the argument prediction parameter; γ is the corner mark prediction parameter; S5: for the aviation accident event triplet feature value of the text data in step S41, set the error output threshold; when the total loss value L of the text data output in step S42 total When the error is greater than or equal to the error output threshold, the weight matrix parameters and bias item parameters of the classifier model in step S2 are updated using the back propagation algorithm, and the execution is continued by returning to step S2; when the total loss value L of the text data output in step S42 is total When it is lower than the error output threshold, the aviation accident event triplet in step S41 is output as the extracted aviation accident report text data.

2. The aviation accident text data extraction method based on cyclic BERT and corner marking according to claim 1 is characterized in that: In step S1, text data feature vectors are embedded, the embedded vectors of the text data are encoded into a label sequence, and the trigger word positions of the text data are embedded and labeled, specifically: S11: embedding the text data of the aviation accident report into a feature vector; embedding the text data into entity words, segmentation embedding and position embedding to form the embedding vector of the text data as follows: x i =e i +p i +s i Among them, x i is the embedding vector of text data; e i is the entity word embedding vector of text data; p i is the segment embedding vector of text data; i Embedding vectors for the positions of text data; S12: Encode the embedding vector of the text data in step S11 into a label sequence to implement the training of entity words, trigger words and context information in the text data of the aviation accident report: h i =BERT(x i ) Among them, h i is the vector of the context information of the i-th trigger word; BERT is the multi-layer Transformer hidden function mapping; S13: The trigger word position embedding annotation of the text data of the aviation accident report; when BERT is used for processing for the first time, the position embedding mark in the text data is 0, and after processing with multiple layers of hidden units, the trigger words and trigger word positions are identified; when BERT is used for processing for the second time, the trigger word position embedding mark in the text data is 1.

3. The aviation accident text data extraction method based on cyclic BERT and corner marking according to claim 1 is characterized in that: In step S2, the double-loop BERT is used to jointly represent the feature vectors, and the probability of the text data argument vector is calculated through the classifier model, specifically: S21: Use the double-loop BERT model to jointly represent the feature vectors, and obtain the joint feature vector of the text data argument vector as follows: J i ={B1(w1+e1),…,B i (w i +e i ),...B m (w m +e m )} Among them, J i is the i-th joint feature vector of the text data argument vector; B i is the i-th embedding vector of the context information of the text data; w i is the i-th word vector of the text data; m is the total number of joint feature vectors of the text data argument vectors; S22: The probability of determining the text data argument vector through the classifier model is: in, Combine the argument head word probability for text data; Combine the argument tail word probability for text data; The weight of the head word of the text data argument; is the weight of the tail word of the text data argument; J s is the joint feature vector of the head word of the text data argument; J e is the joint feature vector of the tail words of the text data argument; B is the embedding vector of the context information of the text data; w is the word vector of the text data; e is the embedding vector of the entity word of the text data.

4. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: The decoding feature marks in step S41 include: a first decoding feature mark HB-TB, a second decoding feature mark HB-TE, a third decoding feature mark HE-TE, a fourth decoding feature mark Minus and a fifth decoding feature mark Null; The first decoding feature mark HB-TB is decoded from the character corresponding to the start position of the subject word to the character corresponding to the start position of the object word; The second decoding feature mark HB-TE is decoded from the character corresponding to the start position of the subject word to the character corresponding to the end position of the object word; The third decoding feature mark HE-TE is decoded from the character corresponding to the end position of the subject word to the character corresponding to the end position of the object word; The fourth decoding feature mark Minus is to decode in sequence according to the subword position order; The fifth decoding feature mark Null is a word that does not need to be decoded and output.

5. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: The trigger word prediction loss value L in step S42 trigger , the trigger word loss value is calculated by the cross entropy loss function: Among them, L trigger is the prediction loss value of the trigger word; N is the number of trigger word samples; C is the number of trigger word categories; is the real label of the trigger word; is the prediction probability of the trigger word; c is the category number of the trigger word.

6. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: The argument prediction loss value L in step S42 arg , specifically: The argument prediction loss model is used to measure aviation accident event arguments. The equations for the predicted loss value at the beginning of the argument and the predicted loss value at the end of the argument are as follows: in, Predict the loss value for the starting position of the argument; Predict the loss value for the end position of the argument; is the real label of the starting position of the argument; Predict the probability for the starting position of the argument; is the actual label of the end position of the argument; Predict the probability of the end position of the argument; Combining the predicted loss value of the starting position and the predicted loss value of the ending position of each argument, the total predicted loss value of the aviation accident event argument is obtained as follows: Among them, A is the set of all arguments; L arg Predict the loss value for the argument.

7. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: The corner mark prediction loss value L in step S42 rel Specifically, the corner label matrix prediction loss is used to measure the difference between the corner label matrix extracted by the double-loop BERT model and the actual result. The specific formula is: Among them, L rel is the prediction loss value of the corner mark; K is the number of relationship categories; is the true label of the k-th category relationship between the i-th entity and the j-th entity; is the predicted probability of the corner labeling matrix; k is the relationship category number.

8. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: In step S5, the error output threshold is set, specifically: S511: Calculate the error term of the output layer of the classifier model: Among them, δ(L total ) is the error term of the output layer of the classifier model; is the total loss function L total The weighted input z(L total )’s partial derivative; is the predicted probability of the classifier model; y is the true label of the classifier model; S512: Calculate the error term of the hidden layer of the classifier model: Among them, δ(l) is the error term of the lth layer of the classifier model; W(l+1) is the weight of the l+1th layer of the classifier model; δ(l+1) is the error value of the l+1th layer of the classifier model; is the element-wise multiplication calculation; g′(z(l)) is the derivative of the activation function; S513: Construct the error output threshold Δ as: Among them, Δ is the error output threshold, which is specifically the error ratio between the loss function difference of the current layer and the output layer and the loss function value of the output layer. When Δ≥50%, the current loss function error does not meet the requirements, and gradient calculation is required to update the classifier model parameters.

9. The method for extracting aviation accident text data based on cyclic BERT and corner marking according to claim 1, characterized in that: In step S5, the back propagation algorithm is used to update the weight matrix parameters and bias item parameters of the classifier model in step S2, specifically: S521: Calculate the total loss function L total The gradient is: in, is the loss function L total The partial derivative of the weight matrix W(l) of the lth layer; W(l) is the weight matrix of the lth layer of the classifier model; a(l-1) is the activation value of the (l-1)th layer of the classifier model, representing the output of the neurons in this layer; is the loss function L total The partial derivative of the bias term b(l) of the lth layer; b(l) is the bias of the lth layer of the classifier model; S522: Update the classifier model parameters using the gradient descent optimization algorithm: Among them, W(l) is the weight matrix of the lth layer of the classifier model; b(l) is the bias of the lth layer of the classifier model; α is the set learning rate, which is used to control the step size of parameter update.

Images