A deep learning-based multi-feature Chinese entity relation extraction method

Abstract

The application discloses a multi-feature Chinese entity relation extraction method based on deep learning, and particularly relates to the technical field of natural language processing, and comprises the following steps: completing multi-feature Chinese word embedding, using BERT to learn character vector, splicing part-of-speech tags and character information position information as word embedding vector input, and sending the word embedding vector input to a multi-feature recurrent convolutional network, the neural network contains Chinese sentence-level features and character-level features, and after a maximum pooling layer, the features are sent to a softmax classifier as final classification vectors; for each sentence, the category corresponding to the value of the maximum probability is the classification result. The application is suitable for Chinese text relation extraction and can effectively deal with the complex relations of Chinese corpus.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a multi-feature Chinese entity relation extraction method based on deep learning. BACKGROUND

[0002] Relation extraction is to extract the relations between entities in the text based on named entity recognition, and form a triple of <entity1, relation, entity2>.

[0003] Early relation extraction is based on rules, and then based on machine learning. The relation extraction based on machine learning can be divided into supervised, semi-supervised, unsupervised, open-domain oriented, and remote supervised entity relation extraction. At present, the rise of deep learning and the development of natural language processing technology and big data technology have brought significant improvement to relation extraction. At present, relation extraction is mainly applied in the construction of knowledge graph, automatic question answering system and mass text translation.

[0004] However, relation extraction still faces many challenges. The actual situation of Chinese text description is much more complex than the simple example sentences in the test. For example, the multiplicity and ambiguity of the relations between entities. At the same time, the artificially annotated test corpus is very limited, while the neural network model needs to be close to the thousands of relation entity triples in reality. Finally, the existing neural network model performs worse in Chinese corpus than in English corpus, which is a difficulty caused by the difference between Chinese grammar and English. SUMMARY

[0005] In order to overcome the above technical defects, the present application provides a multi-feature Chinese entity relation extraction method based on deep learning.

[0006] The technical scheme of the present application is as follows:

[0007] A multi-feature Chinese entity relation extraction method based on deep learning, first uses a BERT model to learn character vectors, and concatenates the part-of-speech tags and character information position information as word embedding vectors. Then the word embedding vectors are input into the multi-feature recurrent convolutional network of the present application. Finally, the input is input into the pooling layer, and the probability is output via the softmax function.

[0008] The method comprises the following steps:

[0009] Step 1: Multi-feature Chinese word embedding: use a BERT model to learn character vectors, and concatenate the character position vectors and part-of-speech tags as word embedding vectors.

[0010] Step 1.1: Use BERT's Tokenizer to segment the text. Each sentence, starting with "CLS" and ending with "SEP", is converted into a token sequence P = {CLS, c1, c2, ..., c...}. n-1 ,c n-1 ,SEP}, sum the token vector, position vector, and sentence vector and input them into BERT.

[0011] Step 1.2: Define the input sentence as a series: W = [w1, w2, ..., w l The final output sequence after BERT encoding is E = [e0, e1, ..., e]. n ,e n+1 ]∈R (n+2) *d T .

[0012] Step 1.3: Chinese contains rich semantic features. In order to resolve word segmentation ambiguity, POSTAG tags are added to add part-of-speech tags to entities.

[0013] Step 1.4: For a Chinese sentence, each character corresponds to a different position, so the boundary feature vector is extracted using a one-hot extraction method.

[0014] Step 2: The above vector concatenation is used as the input of bi-GRU.

[0015] Step 2.1: The GRU has a current input x t The hidden state h passed from the previous node t-1 ,。 Combining x t and h t-1 GRU obtains the output y of the hidden node. t and the hidden state h of the next node t This allows us to obtain two gating states: r for resetting the gate and z for updating the gate.

[0016] r t =σ(w r *[h t-1 ,x t (1)

[0017] z t =σ(w z *[h t-1 ,x t (2)

[0018] Obtain the gating information and use the reset result r t *h t-1 With x t splicing

[0019]

[0020] Update phase, with forgetting and remembering processes

[0021] h t = (1 - z t )*h t-1 + z t *h t (4)

[0022] Finally, the result is output or passed to the next cell

[0023] y t = σ(w o *h t ) (5)

[0024] Step 2.2: Represent the vector from step 1 as e l .

[0025] Step 2.3: Use bi-GRU to get the context representation for each word.

[0026] Step 2.4: The invention defines the left context of w l as c l and the right context as c r . The following are the left and right context representations for w l :

[0027] c l (w l ) = f(w (l) c l (w l-1 ) + w (sl) e(w l-1 ) (6)

[0028] c r (w l ) = f(w (r) c r (w l+1 ) + w (sr) e(w l+1 ) (7)

[0029] w (l) , w (r) convert the hidden layer (context) to the next hidden layer matrix. w (sl) , w (sr) is a matrix used to combine the meaning of the current word with the left or right context of the next word. F i is a nonlinear activation function.

[0030] Step 2.5: The vectors obtained from the above formula (6), formula (7) are processed as follows formula (8):

[0031] w l =[(c l ) T ,(e l ) T ,(c r ) T ] T (8)

[0032] Step 2.6: After the text is represented by the forward and backward scanning of the loop structure, a linear transformation and an activation function are applied to obtain y i (1)

[0033] y i (1) =tanh(w (1) w l +b) (9)

[0034] Step 3: The details of the context are extracted by using a bidirectional recurrent memory network. The vector obtained at the end of step one is input into the Bi-LSTM, and its output is denoted as y i (2) . As a character-level feature.

[0035] Step 4: Step 2 captures the context semantics, and finally concatenates the left and right context vectors and the word embedding vector. Such a model can better eliminate ambiguity than traditional fixed window models, and outputs sentence-level features as a convolutional layer.

[0036] Step 4.1: When calculating all representations of Chinese characters, the application applies a max-pooling layer. The input is the vector of the output of the cyclic convolution and the bidirectional memory network.

[0037] y i (3) =max(y i (1) +y i (2) ) (10)

[0038] Step 4.2: Finally, the softmax function is applied to y i (3) , which can convert the output number into probability

[0039]

[0040] The application has the beneficial effects as follows: the application provides a multi-feature Chinese entity relation extraction method based on deep learning, and the multi-feature recurrent convolutional network of the application contains LSTM, GRU and CNN layers, can extract word-level features and character-level features, is suitable for relation extraction of various Chinese texts, and effectively solves the technical problem of unsatisfactory relation extraction effect. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of the application;

[0042] Figure 2 is a Chinese embedding flowchart of the application;

[0043] Figure 3 is a multi-feature recurrent convolutional network workflow diagram of the application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely in combination with the drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0045] A multi-feature Chinese relation extraction method based on deep learning, as shown in Figure 1 , includes multi-feature Chinese embedding, recurrent convolutional network, max-pooling layer and softmax classifier.

[0046] The specific steps are as follows:

[0047] Step 1: multi-feature Chinese word embedding: using the BERT model to learn character vectors, character position vectors and part-of-speech labels to splice as word embedding vectors. As shown in Figure 2 .

[0048] Step 1-1: using the Tokenizer tokenizer of BERT to perform segmentation operation on the text, each sentence is started with “CLS” and ended with “SEP” structure to convert into Token sequence P={“CLS”,c1,c2,…,c n-1 ,c n-1 ,“SEP”}, summing Token vectors with position vectors and sentence vectors to input BERT.

[0049] Step 1-2: define the input sentence as a series form: W=[w1,w2,…,w l ]. After BERT encoding, the final sequence E=[e0,e1,…,en e n+1 ]∈R (n+2) *d T .

[0050] Step 1-3: Chinese contains rich semantic features, in order to solve the ambiguity of word segmentation, add POSTAG label to add the part of speech of entity.

[0051] Step 1-4: For a Chinese sentence, each word corresponds to a different position, and the boundary feature vector is extracted in one-hot mode.

[0052] Step 2: The above vector is spliced as the input of bi-GRU.

[0053] The input and output of GRU are consistent with traditional recurrent neural network, there is a current input x t and the hidden state h t-1 passed from the last node, which contains the relevant information of the previous node. Combined with x t and h t-1 , GRU gets the output y t of the hidden node and the hidden state h t of the next node. In this way, two gating states can be obtained, r is the reset gate and z is the update gate.

[0054] r t =σ(w r *[h t-1 ,x t ]) (1)

[0055] z t =σ(w z *[h t-1 ,x t ]) (2)

[0056] Get the gating information, use the reset result r t *h t-1 and x t Splicing

[0057]

[0058] Update phase, forgetting and remembering process at the same time

[0059] h t =(1-z t )*h t-1 +z t *h t (4)

[0060] Finally, the result is output or passed to the next cell

[0061] y t = σ(w o *h t ) (5)

[0062] Step 2-1: Represent the vector obtained in step 1 as e l .

[0063] Step 2-2: Use bi-GRU to obtain the context representation for each word. As shown in Figure 3

[0064] Step 2-3: The invention defines the left context of w l as c l and the right context as c r . The following are the left and right context representations for w l :

[0065] c l (w l ) = f(w (l) c l (w l-1 ) + w (sl) e(w l-1 ) (6)

[0066] c r (w l ) = f(w (r) c r (w l+1 ) + w (sr) e(w l+1 ) (7)

[0067] w (l) , w (t) convert the hidden layer (context) to the next hidden layer matrix. w (sl) , w (sr) is a matrix used to combine the semantic meaning of the current word with the left or right context of the next word. F i is a nonlinear activation function.

[0068] Step 2-4: The vectors obtained by the above formula (6), formula (7) are processed as follows formula (8):

[0069] w l = [(c l ) T , (e l ) T , (c r ) T ] T (8)

[0070] ​Step 2-5: After the text is represented, a linear transformation and an activation function are applied to get y i (1)

[0071] y i (1) = tanh(w (1) w l +b) (9)

[0072] Step 3: A bidirectional memory network is used to extract the details of the context better because of its more complex structure and memory gate. The vector obtained at the end of step one is used as the input of the Bi-LSTM, and the output is denoted as y i (2) . As a character-level feature.

[0073] Step 4: Step 2 captures the context semantics, and finally concatenates the left and right context vectors and the word embedding vector. Such a model can better disambiguate than the traditional fixed window, and the sentence-level feature is output as a convolutional layer.

[0074] Step 4-1: When calculating all representations of Chinese characters, the application applies a max-pooling layer. The sum of the output of the cyclic convolution and the vector of the bidirectional memory network is input.

[0075] y i (3) = max(y i (1) +y i (2) ) (10)

[0076] Step 4-2: Finally, the softmax function is applied to y i (3) , which can convert the output number into probability

[0077]

[0078] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A deep learning-based multi-feature Chinese entity relation extraction method, characterized in that, Comprising the following steps: Step 1: Multi-feature Chinese word embedding: learn character vectors using the BERT model, and concatenate character position vectors and part-of-speech tags as word embedding vectors; Step 2: The above vector concatenation is used as the input of bi-GRU; bi-GRU is used as a module for extracting word-level features in the multi-feature recurrent convolutional network; The specific process of step 2 is as follows: Step 2.1: The GRU has a current input x t The hidden state h passed from the previous node t-1 Combined with x t and h t-1 GRU obtains the output y of the hidden node. t and the hidden state h of the next node t This gives us two gate states: r for resetting the gate and z for updating the gate. r t = σ(w r *[h t-1 ,x t ]) (1) z t = σ(w z *[h t-1 ,x t ]) (2) wherein w r , w z represent weight parameters, respectively. Get gating information, using reset result r t h t-1 With x t Splice; What is the weight parameter, is the data containing the current input x t , optionally adding to the current hidden state, that is, preserving the state at the current time; The update phase simultaneously performs the forgetting and memory processes; Finally, the results are output or passed to the next cell; y t = σ(w o *h t ) (5) where σ is a function known as sigmoid, w o denotes a weight parameter; Step 2.2: Represent the vector from step 1 as e l ; Step 2.3: Use bi-GRU to obtain the context representation of each word; Step 2.4: Define w l The left context of c l is c r ; the following are the left and right context representations of w l : c l (w l )=f(w (l) c l (w l-1 )+w (sl) e(w l-1 ) (6) c r (w l )=f(w (r) c r (w l+1 )+w (sr) e(w l+1 ) (7) w (l) , w (r) is a matrix that converts the hidden layer to the next hidden layer;w (sl) , w (sr) is a matrix for combining the semantic meaning of the current word with the left or right context of the next word; f is a non-linear activation function; Step 2.5: The vectors obtained by the above formula (6) and formula (7) are processed as follows formula (8): w l = [(c l ) T , (e l ) T , (c r ) T ] T (8) Step 2.6: The recurrent structure scans the text forward and backward, w l As the text context representation, an activation function tanh is applied, y i (1) As the output of the CNN layer, w (1) As the weight matrix, b is the bias parameter; y i (1) = tanh(w (1) w l +b) (9) where w (1) denotes a weight parameter, b a bias parameter; Step 3: Extract the details of the context using Bi-LSTM, taking the vector obtained at the end of step 1 as the input of Bi-LSTM, and the output is denoted as y i (2) , as a module for extracting character-level features in the multi-feature recurrent convolutional network; Step 4: Step 2 captures the context semantics, and finally concatenates the left and right context vectors and the word embedding vector as the output of the convolutional layer to output the sentence-level features. 2.The deep learning-based multi-feature Chinese entity relation extraction method according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Split the text using the Tokenizer tokenizer of BERT, each sentence starts with CLS and ends with SEP structure into Token sequence P = {CLS, c1, c2, …, c n-1 , SEP}; sum the Token vector and the position vector, the sentence vector to input BERT; Step 1.2: The output of BERT is a sequence of hidden vectors, which are the hidden vectors of each Token in the input sequence. The hidden vector of the CLS Token is the hidden vector of the entire input sequence, which is the input of the next step. Step 1.3: The hidden vector of the CLS Token is input into the classification layer, and the output is the probability of each class. Step 1.2: Define the input sentence as a series form: W = [w1, w2, …, w l ]; encode the final sequence E = [e0, e1, …, e n , e n+1 ] ∈ R (n+2) *d T through BERT; R represents a set of real numbers, d T is the hidden dimension of bert; Step 1.3: Chinese contains rich semantic features, in order to solve the ambiguity of word segmentation, POSTAG tags are added to add the part of speech of entities; Step 1.4: For a Chinese sentence, each character corresponds to a different position, and a one-hot method is used to extract boundary feature vectors. 3.The deep learning based multi-feature Chinese entity relation extraction method according to claim 1, characterized in that, The specific process of step 4 is as follows: Step 4.1: When calculating all representations of Chinese characters, apply max-pooling layer, input recurrent convolution output and bi-directional memory network vector: y i (3) = max(y i (1) + y i (2) ) (10) Step 4.2: Finally, apply the softmax function to y i (3) Convert the output numbers into probabilities:

Images