A pipeline-based approach to extracting government affairs triples

Abstract

A government affairs triple extraction method based on a pipeline solution includes the following steps: 1) obtaining a description text of social conditions and public opinion in a database of a certain government inquiry platform, and predetermining the schema of entities and the schema of relationships; 2) converting the text into a vector and putting it into a BERT model to predict the entity starting boundary; 3) presetting the maximum width of the span, starting from the predicted starting boundary, generating entity spans one by one until the span width is equal to the set maximum width; 4) predicting the entity type of the generated span; 5) pairing the identified entities in pairs to form several sentences, and each entity type is represented by a special token character; 6) replacing the entities in the text with corresponding token characters to obtain a new sentence; 7) putting the new sentence into another BERT model to obtain a sentence vector; 8) concatenating the head and tail entity vectors in the sentence vector, and sending them to a feedforward neural network for relationship prediction, thereby completing the text triple extraction task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence applications, and in particular to a government affairs triplet extraction method based on a pipeline solution. Background Art

[0002] Today's Internet has entered the era of big data. Social networking platforms and communication platforms generate a large amount of video, sound, picture, text and other data every day. How to effectively analyze these data is crucial for the server and the regulator. In actual scenarios, for the server, more accurate big data analysis can provide users with more accurate advertising push services; for the regulator, more accurate big data analysis is conducive to more detailed collection of user behavior and early control of its behavior.

[0003] In the existing technology, the analysis of network platform data is mainly aimed at text data, and the triple extraction of text data is mainly achieved by manual identification, which is inefficient and has a high error rate. With the popularization of natural language processing technology, the triple extraction and analysis process of text data is also realized by deep learning neural network models, but the existing network model structure is relatively complex and the analysis accuracy needs to be improved.

[0004] In summary, in the prior art, the problem of low parsing accuracy still exists in the text type parsing process. Summary of the invention

[0005] In order to improve the efficiency of the current problem of processing triple extraction of government text data, the present invention proposes a government triple extraction method based on a pipeline scheme, which first performs named entity recognition on the text data, and then uses the recognized entities to classify the relationships, thereby completing the task of triple extraction.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A government affairs triple extraction method based on a pipeline scheme includes the following steps:

[0008] 1) Obtain the description text of social conditions and public opinion in the database of a certain political inquiry platform, and pre-set the schema of the entities and relationships that need to be annotated;

[0009] 2) According to the above annotation schema, manually annotate the triple relationship in the description text on the web-based annotation platform, and generate annotated text in a certain format as training data and verification data;

[0010] 3) First, perform named entity recognition on the text data. The process is as follows:

[0011] 3.1) Given a tokenized text X = [x1, x2.......x n ], where x t is the token at position t in the text. After X is input into the BERT model, the hidden representation of the text is obtained as H = [h1,h2...h n ], which represents the process as follows:

[0012] H=BERT(X)

[0013] 3.2) A binary classifier is used to assign a binary label (0 / 1) to each vector to detect whether the object is the starting position of the entity. When the probability of the predicted object exceeds a certain threshold, the object is assigned label 1, otherwise it is assigned label 0. The representation process is:

[0014] P i start_s =σ(W start h i +b start )

[0015] P i start_s represents the probability that the i-th token in the input sequence is recognized as the starting position of the entity, σ is the sigmoid activation function, and W start is the trainable weight, b start It is a deviation;

[0016] 3.3) Preset the maximum width of a span, take the object marked as 1 as the starting position of the entity, and generate span entities in sequence until the width of the span is equal to the maximum width. The representation process of each entity span is:

[0017] h e (s i )=[h START(i) ;h END(i) ;φ(s i )]

[0018] h START(i) is the starting index of span, h END(i) is the end index of span, φ(s i ) is the width feature embedding of span;

[0019] 3.4) Send the entity span generated above into the feedforward neural network to predict the entity type, as shown below:

[0020] P e (e|s i )=softmax(W e he (s i ))

[0021] W e are trainable weights;

[0022] 3.5) The cross entropy loss function for named entity recognition is as follows:

[0023]

[0024] s is a collection of entity spans, s i For the elements in the set, The entity type is gold;

[0025] 4) For the sentences that have completed named entity recognition, the entity spans identified in the sentences are paired in pairs to form span head and tail entity pairs, and several sentences are generated. Each entity type is represented by a special token character;

[0026] 5) Replace the span head and tail entities in the generated sentence with the corresponding entity type token to construct a new sentence;

[0027] 6) Classify the relationship between the head and tail entities in the newly generated sentence. The process is as follows:

[0028] 6.1) Put this new sentence into another BERT model and generate a sentence vector as shown below:

[0029] B={b1,b2......b n}

[0030] 6.2) Take out the corresponding head and tail entity vectors in the sentence vector and concatenate them, as shown below:

[0031] h r (s i ,s j )=[b i ; b j ]

[0032] s i is the head entity, s j is the tail entity, b i is the vector corresponding to the head entity in the sentence, b j is the vector corresponding to the tail entity in the sentence;

[0033] 6.3) The concatenated vectors are fed into a feedforward neural network to predict the relationship, thereby completing the triple extraction task, as shown below:

[0034] P r (r|si ,s j )=softmax(W r h r (s i ,s j ))

[0035] W r are trainable weights;

[0036] 7) The loss function of the relational model training is as follows:

[0037]

[0038] S G is the gold entity collection, Is the gold relationship type for entity pairs.

[0039] The beneficial effects of the present invention are as follows: after preprocessing the text data, the starting boundary of the entity is predicted first, and the entity labeling method of span can solve the problem of entity overlap, greatly improving the accuracy of named entity recognition, and replacing the entity itself with the entity type, making full use of the entity type information, which is of great help in extracting the relationship between the head and tail entities, thereby better completing the task of text triple extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The flowchart of the pipeline-based government triple extraction method is shown. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings.

[0042] Reference Figure 1 ,A government triple extraction method based on a pipeline scheme includes the following steps:

[0043] 1) Obtain the description text of social conditions and public opinion in the database of a certain political inquiry platform, and pre-set the schema of the entities and relationships that need to be annotated;

[0044] 2) According to the above annotation schema, manually annotate the triple relationship in the description text on the web-based annotation platform, and generate annotated text in a certain format as training data and verification data;

[0045] 3) First, perform named entity recognition on the text data. The process is as follows:

[0046] 3.1) Given a tokenized text X = [x1, x2.......x n ], where x tis the token at position t in the text. After X is input into the BERT model, the hidden representation of the text is obtained as H = [h1,h2...h n ], which represents the process as follows:

[0047] H=BERT(X)

[0048] 3.2) A binary classifier is used to assign a binary label (0 / 1) to each vector to detect whether the object is the starting position of the entity. When the probability of the predicted object exceeds a certain threshold, the object is assigned label 1, otherwise it is assigned label 0. The representation process is:

[0049] P i start_s =σ(W start h i +b start )

[0050] P i start_s represents the probability that the i-th token in the input sequence is recognized as the starting position of the entity, σ is the sigmoid activation function, and W start is the trainable weight, b start It is a deviation;

[0051] 3.3) Preset the maximum width of a span, take the object marked as 1 as the starting position of the entity, and generate span entities in sequence until the width of the span is equal to the maximum width. The representation process of each entity span is:

[0052] h e (s i )=[h START(i) ;h END(i) ;φ(s i )]

[0053] h START(i) is the starting index of span, h END(i) is the end index of span, φ(s i ) is the width feature embedding of span;

[0054] 3.4) Send the entity span generated above into the feedforward neural network to predict the entity type, as shown below:

[0055] P e (e|s i )=softmax(W e h e (s i ))

[0056] We are trainable weights;

[0057] 3.5) The cross entropy loss function for named entity recognition is as follows:

[0058]

[0059] s is a collection of entity spans, s i For the elements in the set, is the gold entity type; 4) For the sentences that have completed named entity recognition, pair the entity spans identified in the sentences

[0060] Generate several sentences for the span head and tail entity pairs, and represent each entity type with a special token character;

[0061] 5) Replace the span head and tail entities in the generated sentence with the corresponding entity type token to construct a new sentence;

[0062] 6) Classify the relationship between the head and tail entities in the newly generated sentence. The process is as follows:

[0063] 6.1) Put this new sentence into another BERT model and generate a sentence vector as shown below:

[0064] B={b1,b2......b n}

[0065] 6.2) Take out the corresponding head and tail entity vectors in the sentence vector and concatenate them, as shown below:

[0066] h r (s i ,s j )=[b i ; b j ]

[0067] s i is the head entity, s j is the tail entity, b i is the vector corresponding to the head entity in the sentence, b j is the vector corresponding to the tail entity in the sentence;

[0068] 6.3) The concatenated vectors are fed into a feedforward neural network to predict the relationship, thereby completing the triple extraction task, as shown below:

[0069] P r (r|s i ,s j )=softmax(W r hr (s i ,s j ))

[0070] W r are trainable weights;

[0071] 7) The loss function of the relational model training is as follows:

[0072]

[0073] S G is the gold entity collection, Is the gold relationship type for entity pairs.

[0074] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A government affairs triple extraction method based on a pipeline scheme, characterized in that: The method comprises the following steps: 1) Obtain the description text of social conditions and public opinion in the database of a certain political inquiry platform, and pre-set the schema of the entities and relationships that need to be annotated; 2) According to the above annotation schema, the triple relationship in the description text is manually annotated on the web-based annotation platform, and annotated text in a certain format is generated as training data and verification data; 3) First, perform named entity recognition on the text data. The process is as follows: 3.1) Given a tokenized text X = [x1, x2.......x n ], where x t is the token at position t in the text. After X is input into the BERT model, the hidden representation of the text is obtained as H = [h1,h2...h n ], which represents the process as follows: H=BERT(X) 3.2) A binary classifier is used to assign a binary label (0 / 1) to each vector to detect whether the object is the starting position of the entity. When the probability of the object exceeds a certain threshold, the object is assigned label 1, otherwise it is assigned label 0. The representation process is: P i start_s =σ(W start h i +b start ) P i start_s represents the probability that the i-th token in the input sequence is recognized as the starting position of the entity, σ is the sigmoid activation function, and W start is the trainable weight, b start It is a deviation; 3.3) Preset the maximum width of a span, take the object marked as 1 as the starting position of the entity, and generate span entities in sequence until the width of the span is equal to the maximum width. The representation process of each entity span is: h e (s i )=[h START(i) ;h END(i) ;φ(s i )] h START(i) is the starting index of span, h END(i) is the end index of span, φ(s i ) is the width feature embedding of span; 3.4) Send the entity span generated above into the feedforward neural network to predict the entity type, as shown below: P e (e|s i )=softmax(W e h e (s i )) W e are trainable weights; 3.5) The cross entropy loss function for named entity recognition is as follows: s is a collection of entity spans, s i For the elements in the set, The entity type is gold; 4) For the sentences that have completed named entity recognition, the entity spans identified in the sentences are paired in pairs to form span head and tail entity pairs, and several sentences are generated. Each entity type is represented by a special token character; 5) Replace the span head and tail entities in the generated sentence with the corresponding entity type token to construct a new sentence; 6) Classify the relationship between the head and tail entities in the newly generated sentence. The process is as follows: 6.1) Put this new sentence into another BERT model and generate a sentence vector as shown below: B={b1,b2......b n } 6.2) Take out the corresponding head and tail entity vectors in the sentence vector and concatenate them, as shown below: h r (s i ,s j )=[b i ;b j ] s i is the head entity, s j is the tail entity, b i is the vector corresponding to the head entity in the sentence, b j is the vector corresponding to the tail entity in the sentence; 6.3) The concatenated vectors are fed into a feedforward neural network to predict the relationship, thereby completing the triple extraction task, as shown below: P r (r|s i ,s j )=softmax(W r h r (s i ,s j )) W r are trainable weights; 7) The loss function of the relational model training is as follows: S G is the gold entity collection, Is the gold relationship type for entity pairs.

Images