Chinese entity relation joint extraction method based on deep learning and multi-position pointer labeling

Abstract

The invention discloses a Chinese entity relation joint extraction method based on deep learning and multi-position pointer labeling, and the method comprises the steps: building a joint extraction model based on news text data through employing deep learning technologies, such as a Chinese pre-training model, a bidirectional long-short-term memory network, an attention mechanism, a convolutional neural network, etc. According to the method, multiple positions of the entity relationship are labeled by using multiple position pointer labels, and entities and the relationship among the entities are extracted, so that customers are further mined for an intermediary service platform, and the customer obtaining rate and the working efficiency of the platform are improved.

Description

Technical Field

[0001] The present application belongs to the field of information extraction technology, and specifically relates to a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation. Background Art

[0002] With the advancement of technology and the widespread adoption of the internet, a large number of intermediary service platforms targeting businesses have emerged. These platforms include government procurement contract financing service platforms, small and medium-sized enterprise credit financing service platforms, and investment and wealth management platforms. As intermediary service organizations, competition among these platforms is fierce. In this environment, platforms must not only provide high-quality services but also accurately identify corporate clients and provide high-quality customer resources for partner organizations. Different businesses have different needs and expectations for the platform's services, so the platform must provide products and services that meet these needs. This requires a deep understanding of the specific circumstances of each business. Entity relationship extraction aims to extract valuable information from large amounts of unstructured text data, thereby helping platforms gain a more comprehensive understanding of their customers. Entity relationship extraction provides strong support for intermediary service platforms in understanding their customers.

[0003] Traditional entity relationship extraction mainly involves two methods: pipeline extraction and joint extraction. Pipeline extraction extracts entities and relationships as two independent subtasks, which can easily lead to error accumulation. Joint extraction combines entity extraction and relationship extraction, avoiding the error accumulation problem by increasing the interaction between the two subtasks. Existing joint extraction methods mainly include parameter sharing and sequence labeling. The parameter sharing method shares parameters in entity extraction and relationship extraction, adjusting them together during training to achieve the purpose of interaction between the two subtasks. However, this method cannot solve the problem of complete entity overlap.

[0004] In addition, when extracting entity relationships, the lengths of different types of entities and relationships may vary greatly. Excessively long entities or relationships will affect the effectiveness of model extraction. Summary of the Invention

[0005] The present invention provides a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation, which can annotate the entity location, the relationship type corresponding to the entity, and the entity role, determine the location boundary of the entity relationship, and accurately extract triples.

[0006] The technical solution of this application is as follows: A Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation includes the following steps: S1): Collect news text data about enterprises, associations, and financial service institutions, pre-process the text data on a sentence basis, generate Chinese samples, and divide them into training and test sets; S2): Input Chinese samples into the pre-trained model Bert-base-Chinese to perform word embedding, position embedding, and fragment embedding operations on the Chinese samples to obtain the embedded Chinese word vectors; S3): Input the Chinese word vector into a bidirectional long short-term memory network to capture the long-distance dependency in the sequence data and obtain a word vector that integrates contextual semantics; the bidirectional long short-term memory network includes a forward long short-term memory network and a reverse long short-term memory network; S4): Input the word vector that integrates the contextual semantics into the attention mechanism layer, calculate the weights between words, and generate weighted context representation; use the convolutional neural network to extract features from the word vector sequence; S5): Input the features of the extracted word vector sequence into the fully connected layer to obtain the label prediction output, annotate the output result at the starting position, middle position and end position of the head entity, and calculate the loss of the head entity position annotation; S6): Integrate the head entity position information into the sentence information, calculate the weight of the word vector through the attention mechanism layer, and then extract the word features through the convolutional neural network, and input the extracted features into the fully connected layer; S7): Predict the output relationship type and tail entity label in the fully connected layer, annotate the relationship type and the position of the tail entity in the relationship and tail entity annotation layer, then calculate the loss in the annotation process and extract the triple consisting of the head entity, relationship, and tail entity; S8): Use the pre-trained model to predict the training set and test set, and evaluate the effectiveness of the model with precision, recall, and F1-score values.

[0007] Furthermore, S2) specifically includes the following steps: S2-1) Use WordPiece technology to perform word segmentation on the sentence and obtain its text sequence ; Where, Expressive words, m Indicates the number of words; S2-2) Map word vectors to embedding vectors E ( ) to embed words and embed vectors E ( ) is obtained by looking up the embedding matrix; S2-3) Perform an embedding operation on the word's position vector to obtain the word's position in the sentence. The specific process is as follows: ; ; Where, POS It is the position of the word in the text sequence, starting from 0 by default; j is the dimension index of the embedding vector, The dimension of the embedding vector; S2-4) Perform segment embedding operation using segment embedding vector based on the sentence segment vector; S2-5) Concatenate the embedded word vector, word position vector, and segment vector to obtain the embedded word vector sequence Tok =[ ],in k Expressive words The sentences to which it belongs are as follows: .

[0008] Furthermore, S3) specifically includes the following steps: S3-1) Input the embedded word vector sequence into the forward long short-term memory network and the reverse long short-term memory network to obtain the forward hidden layer output vector and the reverse hidden layer output vector The hidden layer calculation process of the forward LSTM network and the reverse LSTM network is the same. For the forward LSTM network, the specific process of the hidden layer is as follows: ; ; ; ; ; ; Where, 、 、 、 、 、 Represent the input gate, forget gate, output gate, candidate state gate, update gate and output vector of the forward long short-term memory network, respectively. 、 、 and Represent the weight matrix of the corresponding gate, 、 、 、 Indicates bias, σ Represents the activation function sigmoid , Represents vector dot multiplication operation, Represents the output of the hidden layer state at the previous moment; S3-2) Reverse the hidden layer output vector Concatenate with the forward hidden layer output vector , get the word vector that integrates contextual semantics , all word vectors are concatenated to get the output sequence ,in t <m ; The splicing process of the output vector is as follows: .

[0009] Furthermore, S4) specifically includes the following steps: S4-1) Word vector Perform linear transformation to obtain the query vector q , calculated using the scoring function The degree of attention to all hidden states in the sequence is as follows: Where, and represents the weight matrix, represents the weight vector used to generate the score, is the query vector q And each hidden state The score between S4-2) Use softmax The function will score Convert to attention weight , the attention weight The hidden state as a weighted input generates a context vector g The specific process is as follows: ; ; S4-3) Adjust the context vector g The form is To make it suitable for convolution; The specific shape is , is the number of channels, 1 is the height and width; Two layers of convolution are used, and after each layer of convolution, a layer is added ReLu Layer and average pooling layer, output feature map, the specific process is as follows: ; ; ; Where, n ={1,2}, which indicates the order of convolution or pooling operations, is the output of the convolutional layer, is the output of the ReLU layer, is the output of the pooling layer.

[0010] Furthermore, S5) specifically includes the following steps: S5-1) Flattens the result obtained after the second pooling in S4-3) into a one-dimensional vector, inputs the one-dimensional vector into the fully connected layer, uses the softmax function to predict the head entity type label and obtain the label prediction output. The specific process is as follows: ; ; ; Where, y is the output vector of the fully connected layer, expressed as ; k is the number of categories, each Indicates the output of the corresponding category; is the output of flattening the one-dimensional vector, W and b are trainable weight matrices and biases respectively; Indicates Under the condition that the label y Labeled into categories k probability; S5-2) Calculate the position probability of the head entity. When the probability value is greater than the set threshold, the head entity type corresponding to the starting or ending position is marked as 1. When the probability value is within the set threshold range, the head entity type corresponding to the middle position is marked as 1, otherwise it is marked as 0. The head entity position probability value is calculated as follows: ; ; ; Where, 、 and Represented as the first i The probability value of the word being recognized as the starting position, middle position, and end position of the head entity, and Represented as a trainable weight matrix, and Expressed as bias, Represented as the encoded representation of the i-th word in the input sequence; 5-3) Use the binary cross entropy loss function to calculate the loss of the head entity position annotation. The calculation method is as follows: ; ; ; ; Where, 、 and Represent the losses at the starting position, middle position and end position of the head entity respectively, 、 and Represents the start tag, middle tag and end tag respectively. F is the maximum length of the word vector, is the total loss of the head entity.

[0011] Furthermore, S6) specifically includes the following steps: S6-1) Integrate the position information of the head entity into the sentence information in a weighted average manner to obtain a fusion vector sequence corresponding to each head entity ,in c Indicates the order of the head entities; S6-2) Input the fusion vector sequence into the attention mechanism layer to calculate the attention weights between each word, and generate the context representation vector Input to the convolutional neural network, generate a feature map through two convolution and pooling operations, flatten the feature map into a one-dimensional vector, and input it into the fully connected layer.

[0012] Furthermore, S7) specifically includes the following steps: S7-1) is used in the fully connected layer softmaxThe function obtains the predicted output of the relationship type and tail entity position label, and annotates the relationship type and tail entity position. The annotation method is: when the position probability of the tail entity or relationship type sets a threshold, the tail entity or relationship type corresponding to the starting or ending position is marked as 1; when the probability value is within the threshold interval, the tail entity or relationship type corresponding to the middle position is marked as 1, otherwise it is marked as 0; the position probability is calculated as follows: ; ; ; ; ; ; Where, 、 、 、 、 and Represents the first i The probability of a word token being recognized as the starting position of the tail entity, the middle position of the tail entity, the end position of the tail entity, the starting position of the tail entity correspondence, the middle position of the tail entity correspondence, and the end position of the tail entity correspondence, 、 、 、 、 and are all trainable weight matrices, 、 、 、 、 and are all trainable bias matrices, σ for sigmoid Activation function; S7-2) For the head entity, the loss is calculated using the binary cross entropy loss function to calculate the loss of the starting position of the tail entity. , the loss of the middle position of the tail entity , tail entity end position loss , tail entity correspondence starting position loss , the middle position of the tail entity correspondence relationship and tail entity correspondence end position loss , the total loss of the tail entity , the loss of tail entity correspondence And the overall loss is calculated as follows: ; ; ; S7-3) Combine the extracted head entity, the tail entity corresponding to the head entity, and the relationship type corresponding to the tail entity to form a triple. The triple is composed of the head entity head ,relation relation , tail entity tail composition.

[0013] Due to the adoption of the above technical solution, the beneficial effects of this application are as follows: 1. Based on the existing various types of relationships, this application establishes a data foundation for entity relationship extraction on the intermediary service platform for enterprises.

[0014] 2. This application adds a convolutional neural network layer after the attention mechanism layer, which can further extract the features of word vectors, filter out unnecessary information, retain features that are useful for entity and relationship extraction, and further improve the performance of the extraction model.

[0015] 3. This application makes use of the service platform but does not rely on it. It can be applied to other similar service platforms. The extracted triples can be used to build a knowledge graph, which can provide data analysis and decision support for the service platform, helping the service platform understand the needs of enterprises and make personalized business recommendations to enterprises.

[0016] 4. This application is aimed at the joint extraction of entity relationships in a specific field. Taking into account the large difference in the length of different entity relationships in the field, multi-position pointer annotation is performed, which helps to further improve the recognition accuracy, reduce the error rate, and improve the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation provided in this application; Figure 2 The architecture diagram of the Chinese entity relationship joint extraction model provided in this application; Figure 3 A schematic diagram of multiple location annotations of entity relationships in this application. DETAILED DESCRIPTION

[0018] Based on the background technology, through the analysis of the service platform's business data, the service platform's customers come from two aspects: one is cooperative organizations, and the other is enterprises. For enterprises, the personnel who decide whether the enterprise needs intermediary services often hold certain positions in some organizations or associations, and the enterprise may be a member of the organization or association. Therefore, the entities we want to extract here are intermediary service platforms, associations, enterprises, and personnel. The extracted relationships are the relationship between intermediary service agencies and enterprises, the relationship between intermediary service platforms and personnel, the relationship between associations and enterprises, the relationship between associations and personnel, the relationship between enterprises and personnel, and the relationship between enterprises and enterprises.

[0019] As attached Figure 1 As shown, the present application provides a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation, comprising the following steps: S1): Collect news text data about enterprises, associations, and financial service agencies, pre-process the text data in sentence units, form Chinese samples, and divide them into training sets and test sets.

[0020] This embodiment establishes an enterprise association news text dataset by combining the enterprise and association data of the existing service platform with the news text data collected by itself. The dataset contains entities such as enterprises, associations, intermediary service platforms, personnel, and various types of relationships such as intermediary service platforms and enterprises, intermediary service platforms and personnel, associations and enterprises, associations and personnel, enterprises and personnel, and enterprises and enterprises. It provides a relatively comprehensive data foundation for the extraction of entity relationships on the intermediary service platform for enterprises.

[0021] In this implementation, the enterprises and associations that normally log in to the official website are exported through the service platform database and the list of provincial associations. Then, the official websites of the associations, enterprises, and Economic Daily are visited respectively to collect 1,000 news report data such as association news, enterprise news, and intermediary service platform news. The sentence segmentation operation is performed, and the maximum sentence length is set to no more than 100. Sentences with a length of more than 100 are deleted, and meaningless data are removed. Finally, 6,000 data are formed, of which 5,000 are used as training sets and 1,000 are used as test sets.

[0022] S2): Input Chinese samples into the pre-trained model Bert-base-Chinese to perform word embedding, position embedding, and fragment embedding operations on the Chinese samples to obtain the embedded Chinese word vectors; S2) specifically includes the following steps: S2-1) Use WordPiece technology to perform word segmentation on the sentence and obtain its text sequence ; Where, Expressive words, m Indicates the number of words; S2-2) Map word vectors to embedding vectors E ( ) to embed words and embed vectors E ( ) is obtained by looking up the embedding matrix; S2-3) Perform an embedding operation on the word's position vector to obtain the word's position in the sentence. The specific process is as follows: ; ; Where, POS It is the position of the word in the text sequence, starting from 0 by default; j is the dimension index of the embedding vector, The dimension of the embedding vector; S2-4) Perform segment embedding operation using segment embedding vector based on the sentence segment vector; S2-5) Concatenate the embedded word vector, word position vector, and segment vector to obtain the embedded word vector sequence Tok = [ ] ,in k Expressive words The sentences to which it belongs are as follows: .

[0023] S3): Input the Chinese word vector into the bidirectional long short-term memory network to capture the long-distance dependency in the sequence data and obtain the word vector that integrates the contextual semantics; the bidirectional long short-term memory network includes a forward long short-term memory network and a reverse long short-term memory network; the forward and reverse long short-term memory networks are composed of an input gate, a forget gate, a candidate state gate, an update gate and an output gate. The forward one processes the input in the forward order of the time series to capture the semantics of the following context; the reverse one processes the input in the reverse order of the time series to capture the semantics of the previous context. The bidirectional long short-term memory network is a combination of forward and reverse.

[0024] S3) specifically includes the following steps: S3-1) Input the embedded word vector sequence into the forward long short-term memory network and the reverse long short-term memory network to obtain the forward hidden layer output vector and the reverse hidden layer output vector The hidden layer calculation process of the forward LSTM network and the reverse LSTM network is the same. For the forward LSTM network, the specific process of the hidden layer is as follows: ； ； ； ； ； ； Where, 、 、 、 、 、 Represent the input gate, forget gate, output gate, candidate state gate, update gate and output vector of the forward long short-term memory network, respectively. 、 、 and Represent the weight matrix of the corresponding gate, 、 、 、 Indicates bias, σ Represents the activation function sigmoid , Represents vector dot multiplication operation, Represents the output of the hidden layer state at the previous moment; S3-2) Reverse the hidden layer output vector Concatenate with the forward hidden layer output vector , get the word vector that integrates contextual semantics , all word vectors are concatenated to get the output sequence ,in t <m ; The splicing process of the output vector is as follows: .

[0025] S4): Input the word vector that integrates the contextual semantics into the attention mechanism layer, calculate the weights between words, and generate weighted context representation; use the convolutional neural network to extract features from the word vector sequence; S4) specifically includes the following steps: S4-1) Word vector Perform linear transformation to obtain the query vector q , calculated using the scoring function The degree of attention to all hidden states in the sequence is as follows: Where, and represents the weight matrix, represents the weight vector used to generate the score, is the query vector q And each hidden state The score between S4-2) Use softmax The function will score Convert to attention weight , the attention weight The hidden state as a weighted input generates a context vector g The specific process is as follows: ； ； S4-3) Adjust the context vector g to be To make it suitable for convolution; The specific shape is ， is the number of channels, 1 is the height and width; Two layers of convolution are used, and after each layer of convolution, a layer is added ReLu Layer and average pooling layer, output feature map, the specific process is as follows: Where, n={1,2} , which represents the order of convolution or pooling operations, is the output of the convolutional layer, for ReLU The output of the layer, is the output of the pooling layer.

[0026] S5): As attached Figure 3 As shown, the features of the extracted word vector sequence are input into the fully connected layer to obtain the label prediction output, the output result is annotated at the starting position and the end position of the head entity, and the loss of the head entity position annotation is calculated; S5) specifically includes the following steps: S5-1) Flattens the result obtained after the second pooling in S4-3) into a one-dimensional vector, inputs the one-dimensional vector into the fully connected layer, uses the softmax function to predict the head entity type label and obtain the label prediction output. The specific process is as follows: Where, y is the output vector of the fully connected layer, expressed as ; k is the number of categories, each Indicates the output of the corresponding category; is the output of flattening the one-dimensional vector, W and b are trainable weight matrices and biases respectively; Indicates Under the condition that the label y Labeled into categories k probability; S5-2) Calculate the position probability of the head entity. When the probability value is greater than the set threshold, the head entity type corresponding to the starting or ending position is marked as 1. When the probability value is within the set threshold range, the head entity type corresponding to the middle position is marked as 1, otherwise it is marked as 0. The head entity position probability value is calculated as follows: ； ； ； Where, 、 and Represented as the first i The probability value of the word being recognized as the starting position, middle position and end position of the head entity, and Represented as a trainable weight matrix, and Expressed as bias, Represents the first i The encoding representation of each word; In this embodiment, the threshold is set to 0.6, and the threshold range is 0.3-0.6.

[0027] 5-3) Use the binary cross entropy loss function to calculate the loss of the head entity position annotation. The calculation method is as follows: ； ； ； ； Where, 、 and Represent the losses at the starting position, middle position and end position of the head entity respectively, 、 and Represents the start tag, middle tag and end tag respectively. F is the maximum length of the word vector, is the total loss of the head entity.

[0028] S6): Integrate the head entity position information into the sentence information, calculate the weight of the word vector through the attention mechanism layer, and then extract the word features through the convolutional neural network, and input the extracted features into the fully connected layer; S6) specifically includes the following steps: S6-1) Integrate the position information of the head entity into the sentence information in a weighted average manner to obtain a fusion vector sequence corresponding to each head entity ,in c Indicates the order of the head entities; S6-2) Input the fusion vector sequence into the attention mechanism layer to calculate the attention weights between each word, and generate the context representation vector Input to the convolutional neural network, generate a feature map through two convolution and pooling operations, flatten the feature map into a one-dimensional vector, and input it into the fully connected layer; S7): Predict the output relationship type and tail entity label in the fully connected layer, annotate the relationship type and the position of the tail entity in the relationship and tail entity annotation layer, then calculate the loss in the annotation process and extract the triples; S8): Use the pre-trained model to predict the training set and test set, and evaluate the effectiveness of the model with precision, recall, and F1-score values.

[0029] S7) specifically includes the following steps: S7-1) is used in the fully connected layer softmax The function obtains the predicted output of the relationship type and tail entity position label, and annotates the relationship type and tail entity position. The annotation method is: when the position probability of the tail entity or relationship type sets a threshold, the tail entity or relationship type corresponding to the starting or ending position is marked as 1; when the probability value is within the threshold interval, the tail entity or relationship type corresponding to the middle position is marked as 1, otherwise it is marked as 0; the position probability is calculated as follows: Where, 、 、 、 、 and Represents the first i The probability of a word token being recognized as the starting position of the tail entity, the middle position of the tail entity, the end position of the tail entity, the starting position of the tail entity correspondence, the middle position of the tail entity correspondence, and the end position of the tail entity correspondence, 、 、 、 、 and are all trainable weight matrices, 、 、 、 、 and are all trainable bias matrices, σ for sigmoid Activation function; S7-2) For the head entity, the loss is calculated using the binary cross entropy loss function to calculate the loss of the starting position of the tail entity. , the loss of the middle position of the tail entity , tail entity end position loss , tail entity correspondence starting position loss , the middle position of the tail entity correspondence relationship and tail entity correspondence end position loss , the total loss of the tail entity , the loss of tail entity correspondence And the overall loss is calculated as follows: ; ; ; S7-3) Combine the extracted head entity, the tail entity corresponding to the head entity, and the relationship type corresponding to the tail entity to form a triple. The triple is composed of the head entity head ,relation relation , tail entity tail composition.

[0030] Anything not described in this application can be achieved by adopting or drawing on existing technologies. The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation, characterized in that: The following steps are involved: S1): Collect news text data about enterprises, associations, and financial service agencies, pre-process the text data in sentences, form Chinese samples and divide them into training sets and test sets; S2): Input Chinese samples into the pre-trained model Bert-base-Chinese to perform word embedding, position embedding, and fragment embedding operations on the Chinese samples to obtain the embedded Chinese word vectors; S3): Input the Chinese word vector into a bidirectional long short-term memory network to capture the long-distance dependency in the sequence data and obtain a word vector that integrates contextual semantics; the bidirectional long short-term memory network includes a forward long short-term memory network and a reverse long short-term memory network; S4): Input the word vector that integrates the contextual semantics into the attention mechanism layer, calculate the weights between words, and generate weighted context representation; extract features from the word vector sequence through a convolutional neural network; S5): Input the features of the extracted word vector sequence into the fully connected layer to obtain the label prediction output, annotate the output result at the starting position, middle position and end position of the head entity, and calculate the loss of the head entity position annotation; S6): Integrate the head entity position information into the sentence information, calculate the weight of the word vector through the attention mechanism layer, extract the word features through the convolutional neural network, and input the extracted features into the fully connected layer; S7): predict the output relation type and tail entity label in the fully connected layer, annotate the relation type and the position of the tail entity in the relation and tail entity annotation layer, then calculate the loss in the annotation process and extract the triple consisting of the head entity, relation, and tail entity; S8): The training set and test set are predicted using the pre-trained model, and the effectiveness of the model is evaluated using precision, recall, and F1-score values.

2. According to claim 1, a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation is characterized in that: S2) specifically includes the following steps: S2-1) Use WordPiece technology to segment the sentence and obtain its text sequence ; In the formula, Indicative words, m Indicates the number of words; S2-2) Map word vectors to embedding vectors E ( ) to embed words and embed vectors E ( ) is obtained by looking up the embedding matrix; S2-3) Perform an embedding operation on the word position vector to obtain the position of the word in the sentence. The specific process is as follows: ； ； In the formula, POS is the position of the word in the text sequence, starting from 0 by default; j is the dimension index of the embedding vector, The dimension of the embedding vector; S2-4) Perform segment embedding operation using segment embedding vector based on the segment vector of the sentence; S2-5) Concatenate the embedded word vector, word position vector, and segment vector to obtain the embedded word vector sequence Tok =[ ],in k Expressive words The sentences to which it belongs are as follows: 。 3. According to claim 2, a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation is characterized in that: S3) specifically includes the following steps: S3-1) Input the embedded word vector sequence into the forward long short-term memory network and the reverse long short-term memory network to obtain the forward hidden layer output vector and the reverse hidden layer output vector ; The hidden layer calculation process of the forward LSTM network and the reverse LSTM network is the same. For the forward LSTM network, the specific process of the hidden layer is as follows: ； ； ； ； ； ； In the formula, 、 、 、 、 、 Represent the input gate, forget gate, output gate, candidate state gate, update gate and output vector of the forward long short-term memory network, respectively. 、 、 and Respectively represent the weight matrix of the corresponding gate, 、 、 、 Indicates bias, σ Represents the activation function sigmoid , represents the vector dot multiplication operation, Represents the output of the hidden layer state at the previous moment; S3-2) Reverse the hidden layer output vector Concatenate with the forward hidden layer output vector , get the word vector that integrates the context semantics , all word vectors are concatenated to get the output sequence ,in t <m ; The concatenation process of the output vector is as follows: 。 4. According to claim 3, a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation is characterized in that: S4) specifically includes the following steps: S4-1) Word vector Perform a linear transformation operation to obtain the query vector q , using the score function to calculate The degree of attention to all hidden states in the sequence is as follows: In the formula, and represents the weight matrix, represents the weight vector used to generate the score, is the query vector q And each hidden state The score between S4-2) Use softmax The function will score Convert to attention weight , the attention weight As the hidden state of the weighted input, generate the context vector g , the specific process is as follows: ； ； S4-3) Adjust the context vector g The form is to make it suitable for convolution; The specific shape is , is the number of channels, 1 is the height and width; Two layers of convolution are used, and after each layer of convolution, a layer is added ReLu Layer and average pooling layer, output feature map, the specific process is as follows: ； ； ； In the formula, n ={1,2}, which indicates the order of convolution or pooling operations. is the output of the convolutional layer, is the output of the ReLU layer, is the output of the pooling layer.

5. According to claim 4, a Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation is characterized in that: S5) specifically includes the following steps: S5-1) Flattens the result obtained after the second pooling in S4-3) into a one-dimensional vector, inputs the one-dimensional vector into the fully connected layer, uses the softmax function to predict the head entity type label and obtains the label prediction output. The specific process is as follows: ； ； ； In the formula, y is the output vector of the fully connected layer, expressed as ; k is the number of categories, each Represents the output of the corresponding category; is the output of flattening a one-dimensional vector, W and b They are trainable weight matrices and biases respectively; Indicated in Under the condition that y Labeled into categories k probability; S5-2) Calculate the position probability of the head entity. When the probability value is greater than the set threshold, the head entity type corresponding to the start or end position is marked as 1; when the probability value is within the set threshold interval, the head entity type corresponding to the middle position is marked as 1, otherwise it is marked as 0. The calculation method of the head entity position probability value is as follows: ； ； ； In the formula, 、 and Represented as the first i The probability value of the word being recognized as the starting position, middle position, and end position of the head entity, and Represented as a trainable weight matrix, and Expressed as bias, Represented as the encoding representation of the i-th word in the input sequence; 5-3) The binary cross entropy loss function is used to calculate the loss of the head entity position annotation. The calculation method is as follows: ； ； ； ； In the formula, 、 and Respectively represent the loss of the starting position, middle position and end position of the head entity, 、 and Represent the start tag, middle tag and end tag respectively. F is the maximum length of the word vector, is the total loss of the head entity.

6. The Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation according to claim 5 is characterized in that: S6) specifically includes the following steps: S6-1) Integrate the position information of the head entity into the sentence information in a weighted average manner to obtain a fusion vector sequence corresponding to each head entity ,in c Indicates the order of the header entities; S6-2) Input the fused vector sequence into the attention mechanism layer to calculate the attention weights between each word, and generate the context representation vector The input is sent to the convolutional neural network, a feature map is generated through two convolution and pooling operations, the feature map is flattened into a one-dimensional vector, and the input is sent to the fully connected layer.

7. The Chinese entity relationship joint extraction method based on deep learning and multi-position pointer annotation according to claim 6 is characterized in that: S7) specifically includes the following steps: S7-1) Used in the fully connected layer softmax The function obtains the predicted output of the relationship type and tail entity position label, and marks the position of the relationship type and tail entity. The marking method is: when the position probability of the tail entity or relationship type sets a threshold, the tail entity or relationship type corresponding to the start or end position is marked as 1; when the probability value is within the threshold interval, the tail entity or relationship type corresponding to the middle position is marked as 1, otherwise it is marked as 0; the position probability is calculated as follows: ； ； ； ； ； ； In the formula, 、 、 、 、 and Respectively represent the first i The probability of identifying a word token as the starting position of the tail entity, the middle position of the tail entity, the end position of the tail entity, the starting position of the tail entity correspondence, the middle position of the tail entity correspondence, and the end position of the tail entity correspondence, 、 、 、 、 and are all trainable weight matrices, 、 、 、 、 and are all trainable bias matrices, σ for sigmoid Activation function; S7-2) For the head entity, the loss is calculated using the binary cross entropy loss function to calculate the loss of the starting position of the tail entity. , the loss of the middle position of the tail entity , tail entity end position loss , tail entity correspondence starting position loss , the middle position of the tail entity corresponding relationship The end position loss of the corresponding relationship with the tail entity , the total loss of the tail entity , the loss of the tail entity correspondence And the overall loss is calculated as follows: ； ； ； S7-3) Combining the extracted head entity, the tail entity corresponding to the head entity, and the relationship type corresponding to the tail entity to form a triple, wherein the triple is composed of the head entity head ,relation relation , tail entity tail composition.

Images