Named Entity Recognition Method for Chinese Legal Documents Based on Enhanced Sequence Features

Abstract

The present invention discloses a method for identifying named entities in Chinese legal documents based on enhanced sequence features, including: treating sentences in the legal document to be identified as character sequences, extracting the three features of pinyin, glyph and strokes of each character in the sentence to obtain a character feature vector, fusing the character feature vector with a pre-trained dictionary to obtain an enhanced character feature vector; using the enhanced character feature vector to represent potential words in the character sequence, and identifying the word position information and length; setting a fusion filter F to filter and fuse words of various lengths to obtain feature vectors of the same length; using an attention mechanism to integrate character-level features into the above, thereby obtaining a feature vector of the fused word; inputting the feature vector of the fused word into an improved Transformer legal document entity extraction model to extract the entity feature sequence in the legal document to be identified. The present invention greatly improves the accuracy of entity recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for identifying named entities in Chinese legal documents based on enhanced sequence features. Background Art

[0002] In recent years, with the rapid development of computer networks and artificial intelligence technology, natural language processing has become an important method for processing massive text information. As the core of natural language processing tasks, named entity recognition is of great significance in improving the accuracy and reliability of text recognition. However, traditional character-based neural networks have inherent defects in entity extraction, that is, they cannot fully utilize the implicit feature information of Chinese characters themselves, which greatly affects the entity recognition effect.

[0003] Character Embedding (CE) is a common method for processing Chinese text information. It converts Chinese characters into vector form according to the dictionary, so that it can be used as the input of the neural network. The existing CE method has certain advantages in the application scenario of entity fuzzy extraction. Liu K et al. fused the features of the two levels of characters and words through an improved CNN model. P. Das et al. used an unsupervised method based on a graph clustering algorithm to extract entity relationships in the corpus, which can effectively classify entities. However, in an environment where the entity types are complex and there are dependencies between entities, it is difficult for existing embedding mechanisms to achieve relatively ideal results. The main reason is that a single character embedding cannot take into account the feature information of the sentence context, resulting in the loss of semantic information.

[0004] Many researchers at home and abroad have conducted research on the above problems. Dong XY et al. used the self-attention mechanism to fuse the features of characters and words and input them into the model, which improved the accuracy of entity extraction. Li F et al. proposed a dynamic meta-embedding method that can achieve good results in specific tasks, but its domain transferability is poor. However, the above methods still have shortcomings: (1) Most current character embedding methods only use pre-trained dictionaries, while ignoring the characteristic information of Chinese characters themselves. (2) In the process of word embedding, there will be a labeling conflict problem where one character corresponds to multiple words; (3) Due to the high complexity of the model, its training time is long and the model cannot be updated frequently. Summary of the invention

[0005] In response to the problems of complex relationships, strong vocabulary expertise and low recognition rate in Chinese legal documents, the present invention proposes a Chinese legal document named entity recognition method based on enhanced sequence features (FSLTM, Feature Subsequencebased Lattice-Transformer Model), which greatly improves the accuracy of entity recognition.

[0006] To achieve the above objectives, this application proposes a method for named entity recognition of Chinese legal documents based on enhanced sequence features, including:

[0007] The sentence S in the legal document to be recognized is S = [c1, c2, ..., c n-1 ,c n ] is processed as a character sequence, and for each character c in the sentence i The pinyin, shape and strokes of the word are extracted to obtain a word feature vector, and the word feature vector is integrated with the pre-trained dictionary to obtain an enhanced word feature vector Using the enhanced character feature vector, potential words in the character sequence are represented and the word position information and length are identified;

[0008] Set the fusion filter F to filter and fuse words of different lengths to obtain feature vectors of the same length. Use the attention mechanism to incorporate character-level features into the feature vector Then we get the feature vector of the fused word

[0009] The feature vector of the fused word Input into the improved Transformer legal document entity extraction model to extract the entity feature sequence in the legal document to be identified

[0010] Furthermore, for the newly added legal documents, after inputting them into the improved Transformer legal document entity extraction model, the newly appearing words are extracted. As the input of the phantom space model PSM, the corresponding prediction feature sequence is obtained

[0011] The entity feature sequence And the newly added prediction feature sequence They are input together into the conditional random field layer CRF to obtain the predicted label result.

[0012] Furthermore, for each character c in the sentence i The pinyin, glyph and stroke features of the character are extracted to obtain the character feature vector, specifically: the pinyin, glyph and stroke features of the character are extracted to obtain the corresponding feature coding table; the pinyin sub-vector is obtained according to the feature coding table Glyph subvector and stroke sub-vector Character vector

[0013] Furthermore, the word feature vector is fused with the pre-trained dictionary to obtain an enhanced word feature vector Specifically: According to the pre-trained dictionary, get the character c i The corresponding word embedding vector Embed the word into a vector Character vector with word Merge to get enhanced word feature vector

[0014] Further, the enhanced character feature vector is used to represent the potential words in the character sequence, and the word position information and length are identified, specifically: the potential words in the character sequence are represented as Where h represents the subscript of the character corresponding to the starting position of the word, t represents the subscript of the character corresponding to the ending position of the word, and L represents the length of the word, so L = th.

[0015] Furthermore, a fusion filter F is set to filter and fuse words of different lengths to obtain feature vectors of the same length. At the same time, the attention mechanism is used to integrate character-level features into the feature vector Then we get the feature vector of the fused word Specifically:

[0016] First, the window size of the fusion filter F is set to 2. For words of different lengths, the softmax function is used to filter and fuse them to obtain the feature vectors of words of the same length. in L-1 [*,m:m+1],F L-1 > k is the Kronecker product operation; S L-1 represents a sequence of words with a length of L-1, [*,m,m+1] represents the calculation of every two adjacent columns, m represents the current number of columns, the range is [0,d-1], d is the number of word vectors of this length, b L-1 It indicates the offset of word length L-1, and the superscript K is the representation of Kronecker product operator;

[0017] Use the attention mechanism to incorporate character-level features into the feature vector Get the feature vector of the fused word in and is the Hadamard product operation; is the input control vector, is the characteristic control vector, u m The output control vector is: ​

[0018] in

[0019]

[0020] W i is the input weight matrix, W f is the feature weight matrix, b i is the input offset compensation, b f is the characteristic offset compensation, U i is the input attention weight matrix, U f is the feature attention weight matrix. Represents a potential word of length L-1.

[0021] Furthermore, the improved Transformer legal document entity extraction model includes an encoder layer, a decoder layer, and an attention layer;

[0022] The feature vector of the fused word The encoder layer includes a shallow pre-trained CNN network and two layers of Bi-LSTM neural network. A residual connection is established between the output of the CNN network and the output of the last layer of Bi-LSTM neural network to obtain the output feature vector of the encoder layer. The CNN network is used to extract the basic features of the fused word vectors, and the Bi-LSTM network is used to extract the context information of the fused word vectors;

[0023] At the decoder layer, the predefined label sequence is vectorized and embedded, and then the information is extracted through a two-layer LSTM neural network to obtain the output feature vector of the decoder layer. Then the output feature vectors of the encoder layer and the decoder layer are concatenated and added with the calculated value p of their position encoding. n , together as the input of the attention layer in

[0024] In the attention layer, the residual attention mechanism is used to learn the weight distribution coefficient, strengthen the focus on key words, and further capture the sequence characteristics of the input text. At the same time, parallel computing is used to shorten the model training time; the attention layer outputs the feature information vector

[0025] Furthermore, for the newly added legal documents, after inputting them into the improved Transformer legal document entity extraction model, the newly appearing words are extracted. As the input of the phantom space model PSM, the corresponding prediction feature sequence is obtained Specifically:

[0026] Compare with the vocabulary list to filter out new words that appear in sentences in legal documents and using the new words as input of a phantom space model PSM;

[0027] In the Phantom Space Model (PSM), we obtain the meaning of new words from similar words. Among them, select the one that contains the most meaningful (highest similarity) information As a new word Approximate replacement for ;

[0028] Split the newly added legal document sentences into a collection of words And according to the representative information contained in the new word Get the embedding vector of the new word As the output of the phantom space model PSM;

[0029] The embedding vector of the new word Input into the improved Transformer legal document entity extraction model. The implementation process is the same as the input vector of the fused word feature. The output result of the improved Transformer legal document entity extraction model is the predicted feature sequence corresponding to the new word.

[0030] Further, the entity feature sequence And the newly added prediction feature sequence They are input into the conditional random field layer CRF together and the predicted label results are output, specifically:

[0031] Entity Feature Sequence And the newly added prediction feature sequence Perform Kronecker product operation to obtain the fused feature sequence M n ;

[0032] The fused feature sequence M n Input to the conditional random field layer CRF for label classification to generate the optimal label sequence Predict_label = [tag1, t0g2, ..., tag n-1 , tag n ].

[0033] The above technical solution adopted by the present invention has the following advantages compared with the prior art: This method can fully explore the implicit information of words and phrases based on feature subsequences, and use the improved Transformer model to process feature information, which can greatly improve the accuracy of entity recognition in legal documents in the judicial field. At the same time, a phantasm space model PSM for dynamic update is proposed, which extracts content using synonyms of new words in the judicial corpus, can handle newly emerging professional vocabulary, and realizes the dynamic expansion of vocabulary. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a Chinese legal document named entity recognition method based on enhanced sequence features. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0036] Embodiment 1

[0037] As Figure 1 shown, the present application provides a Chinese legal document named entity recognition method based on enhanced sequence features, specifically including:

[0038] S1: Process the sentence S = [c1, c2,..., c n-1 , c n in the legal document to be recognized as a character sequence, extract the three features of the pinyin, glyph, and stroke of each character c i in the sentence to obtain a character feature vector, and fuse the pre-trained dictionary to obtain an enhanced character feature vector Use the enhanced character feature vector to represent the potential words in the character sequence, and identify the word position information and length;

[0039] Specifically, the present invention focuses on extracting the features of the pinyin, glyph, and stroke of Chinese characters, and obtains the corresponding feature encoding tables respectively, as shown in Tables 1 to 3. According to the feature encoding tables, the pinyin sub-vector glyph sub-vector and stroke sub-vector can be obtained. As shown in the table, encoding the Chinese character "囚", the result is "qiu1G58674", where "qiu1" is its pinyin feature "G" is its glyph feature "58674" is its stroke feature Arrange the above features to get the word feature vector

[0040] Table 1 Chinese character pinyin feature coding table

[0041]

[0042] Table 2 Chinese character font feature coding table

[0043]

[0044] Table 3 Chinese character stroke feature coding table

[0045]

[0046] According to the pre-trained dictionary, get the character c i The corresponding word embedding vector Embed the word into a vector Character vector with word Merge to get enhanced word feature vector

[0047] Represent potential words in a character sequence as Where h represents the subscript of the character corresponding to the starting position of the word, t represents the subscript of the character corresponding to the ending position of the word, and L represents the length of the word, so L = th.

[0048] S2: Set the fusion filter F to filter and fuse words of different lengths (words are regarded as word vectors with a length of 1) to obtain feature vectors of the same length Use the attention mechanism to incorporate character-level features into the feature vector Then we get the feature vector of the fused word

[0049] Specifically, first set the window size of the fusion filter F to 2, and use the softmax function to filter and fuse word vectors of different lengths (words are regarded as word vectors of length 1) to obtain feature vectors of words of the same length. in L-1 [*, m:m+1], F L-1 > K is the Kronecker product operation; S L-1 represents a sequence of words with a length of L-1, [*,m,m+1] represents the calculation of every two adjacent columns, m represents the current number of columns, the range is [0,d-1], d is the number of word vectors of this length, b L-1 It indicates the offset of word length L-1, and the superscript K is the representation of Kronecker product operator; ​

[0050] Use the attention mechanism to incorporate character-level features into the feature vector Get the feature vector of the fused word in and is the Hadamard product operation; is the input control vector, is the characteristic control vector, u m The output control vector is:

[0051] in

[0052]

[0053] W f is the input weight matrix, W f is the feature weight matrix, b i is the input offset compensation, b f is the characteristic offset compensation, U i is the input attention weight matrix, U f is the feature attention weight matrix. Represents a potential word of length L-1.

[0054] S3: feature vector of the fused word Input into the improved Transformer legal document entity extraction model to extract the entity feature sequence in the legal document to be identified

[0055] Specifically, the improved Transformer legal document entity extraction model includes an encoder layer, a decoder layer, and an attention layer;

[0056] Encoder layer: The feature vectors of the fused words The encoder layer includes a shallow pre-trained CNN network and two layers of Bi-LSTM neural network. A residual connection is established between the output of the CNN network and the output of the last layer of Bi-LSTM neural network to obtain the output feature vector of the encoder layer. The CNN network is used to extract the basic features of the fused word vectors, and the Bi-LSTM network is used to extract the context information of the fused word vectors;

[0057] Decoder layer: The predefined label sequence is vectorized and embedded, and then the information is extracted through two layers of LSTM neural network to obtain the output feature vector of the decoder layer. Then the output feature vectors of the encoder layer and the decoder layer are concatenated and added with the calculated value p of their position encoding. n, together as the input of the attention layer in

[0058] Attention layer: Use the residual attention mechanism to learn the weight distribution coefficient, strengthen the focus on key words, and capture the sequence characteristics of the input text At the same time, parallel computing is used to shorten the model training time; the attention layer outputs the feature information vector

[0059] S4: For newly added legal documents, after inputting them into the improved Transformer legal document entity extraction model, the newly appearing words are extracted. As the input of the phantom space model PSM, the corresponding prediction feature sequence is obtained

[0060] Specifically, compare with the vocabulary list to filter out new words that appear in sentences in legal documents and taking the new words as input of a phantom space model PSM to process them;

[0061] The meanings of new words are collected from online dictionaries, Wikipedia, web searches, and other websites, and the words they represent are checked to see if they exist in the existing embedding model; when the model does not embed certain words, they are replaced by another expression through the PSM model. For the PSM model, it is first necessary to select words with similar meanings to the new words from similar lexicons as approximate substitutes for the new words. Since the present invention processes legal documents in the judicial field, the legal terms that appear therein contain information that is similar to existing legal terms, so it is possible to select words from similar lexicons. Among them, select the one that contains the most meaningful (highest similarity) information As a new word An approximate replacement for .

[0062] Split the newly added legal document sentences into a collection of words And according to the representative information contained in the new word Get the embedding vector of the new word As the output of the phantom space model PSM;

[0063] The embedding vector of the new word Input to the improved Transformer legal document entity extraction model. The implementation process is the same as the input vector of the fused word feature. The output result of the improved Transformer legal document entity extraction model is the predicted feature sequence corresponding to the new word.

[0064] S5: Sequence entity features and the predicted feature sequence of the new text They are input into the CRF layer together and the predicted label results are output.

[0065] Entity Feature Sequence And the newly added prediction feature sequence Perform Kronecker product operation to obtain the fused feature sequence M n ;

[0066] The fused feature sequence M n Input to the conditional random field layer CRF for label classification to generate the optimal label sequence Predict_label = [tag1, tag2, ..., tag n-1 , tag n The CRF layer is used for character-level sequence annotation, which can automatically learn constraints when training data to ensure that the final prediction result is optimal.

[0067] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A named entity recognition method for Chinese legal documents based on enhanced sequence features, characterized in that: include: The sentence S in the legal document to be recognized is S = [c1, c2, ..., c n-1 ,c n ] is processed as a character sequence, and for each character c in the sentence i The pinyin, shape and strokes of the word are extracted to obtain a word feature vector, and the word feature vector is fused with a pre-trained dictionary to obtain an enhanced word feature vector Using the enhanced character feature vector, potential words in the character sequence are represented and the word position information and length are identified; Set the fusion filter F to filter and fuse words of different lengths to obtain feature vectors of the same length. Use the attention mechanism to incorporate character-level features into the feature vector Then we get the feature vector of the fused word Specifically, first set the window size of the fusion filter F, and use the softmax function to filter and fuse words of different lengths to obtain the feature vectors of words of the same length. in L-1 [*,m:m+1],F L-1 > K is the Kronecker product operation; S L-1 represents a sequence of words with a length of L-1, [*,m,m+1] represents the calculation of every two adjacent columns, m represents the current number of columns, the range is [0,d-1], d is the number of word vectors of this length, b L-1 It indicates the offset of word length L-1, and the superscript K is the representation of Kronecker product operator;​ Use the attention mechanism to incorporate character-level features into the feature vector Get the feature vector of the fused word in and is the Hadamard product operation; is the input control vector, is the characteristic control vector, u m The output control vector is: in W i is the input weight matrix, W f is the feature weight matrix, b i is the input offset compensation, b f is the characteristic offset compensation, U i is the input attention weight matrix, U f is the feature attention weight matrix; Represents potential words of length L-1; The feature vector of the fused word Input into the improved Transformer legal document entity extraction model to extract the entity feature sequence in the legal document to be identified The improved Transformer legal document entity extraction model includes an encoder layer, a decoder layer, and an attention layer; The feature vector of the fused word The encoder layer includes a shallow pre-trained CNN network and two layers of Bi-LSTM neural network. A residual connection is established between the output of the CNN network and the output of the last layer of Bi-LSTM neural network to obtain the output feature vector of the encoder layer. The CNN network is used to extract the basic features of the fused word vectors, and the Bi-LSTM network is used to extract the context information of the fused word vectors; At the decoder layer, the predefined label sequence is vectorized and embedded, and then the information is extracted through a two-layer LSTM neural network to obtain the output feature vector of the decoder layer. Then the output feature vectors of the encoder layer and the decoder layer are concatenated and added with the calculated value p of their position encoding. n , together as the input of the attention layer in In the attention layer, the residual attention mechanism is used to learn the weight distribution coefficient, strengthen the focus on key words, and capture the sequence characteristics of the input text. At the same time, parallel computing is used to shorten the model training time; the attention layer outputs the feature information vector 2. According to claim 1, the method for identifying named entities in Chinese legal documents based on enhanced sequence features is characterized in that: For the newly added legal documents, after inputting them into the improved Transformer legal document entity extraction model, the newly appearing words are extracted. As the input of the phantom space model PSM, the corresponding prediction feature sequence is obtained The entity feature sequence And the newly added prediction feature sequence They are input together into the conditional random field layer CRF to obtain the predicted label result.

3. According to the method for named entity recognition of Chinese legal documents based on enhanced sequence features as described in claim 1, it is characterized in that: For each character c in the sentence i The pinyin, glyph and stroke features of the character are extracted to obtain the character feature vector, specifically: the pinyin, glyph and stroke features of the character are extracted to obtain the corresponding feature coding table; the pinyin sub-vector is obtained according to the feature coding table Glyph subvector and stroke sub-vector Character vector 4. According to claim 1, the method for identifying named entities in Chinese legal documents based on enhanced sequence features is characterized in that: The word feature vector is fused with the pre-trained dictionary to obtain an enhanced word feature vector Specifically: According to the pre-trained dictionary, get the character c i The corresponding word embedding vector Embed the word into a vector Character vector with word Merge to get enhanced word feature vector 5. According to the method for named entity recognition of Chinese legal documents based on enhanced sequence features as described in claim 1, it is characterized in that: The enhanced word feature vector is used to represent the potential words in the character sequence, and the word position information and length are identified, specifically: the potential words in the character sequence are represented as Where h represents the subscript of the character corresponding to the starting position of the word, t represents the subscript of the character corresponding to the ending position of the word, and L represents the length of the word, so L = th.

6. According to claim 2, the method for identifying named entities in Chinese legal documents based on enhanced sequence features is characterized in that: For the newly added legal documents, after inputting them into the improved Transformer legal document entity extraction model, the newly appearing words are extracted. As the input of the phantom space model PSM, the corresponding prediction feature sequence is obtained Specifically: Compare with the vocabulary list to filter out new words that appear in sentences in legal documents and using the new words as input of a phantom space model PSM; In the Phantom Space Model (PSM), we obtain the meaning of new words from similar words. Select the one that contains the most meaningful information. As a new word Approximate replacement for ; Split the newly added legal document sentences into a collection of words And according to the representative information contained in the new word Get the embedding vector of the new word As the output of the phantom space model PSM; The embedding vector of the new word Input to the improved Transformer legal document entity extraction model, the output result of the improved Transformer legal document entity extraction model is the predicted feature sequence corresponding to the new word 7. The method for identifying named entities in Chinese legal documents based on enhanced sequence features according to claim 6, characterized in that: The entity feature sequence And the newly added prediction feature sequence They are input into the conditional random field layer CRF together and the predicted label results are output, specifically: Entity Feature Sequence And the newly added prediction feature sequence Perform Kronecker product operation to obtain the fused feature sequence M n ; The fused feature sequence M n Input to the conditional random field layer CRF for label classification to generate the optimal label sequence Predict_label = [tag1, tag2, ..., tag n-1 ,tag n ].

Images